HTTP ERROR: 503 Problems accesing /opennms/index.jsp. Reason: service unavailable

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

HTTP ERROR: 503 Problems accesing /opennms/index.jsp. Reason: service unavailable

javiergf
Hi,
I'd really appreciate if someone could give me some help on this error,
we are running OpenNMS 1.9.94 and we did the system up & running for over a month and now we always get this error when we try access to OpenNMS. What is extremely weird is that none of the configuration files have been touched and if you stop and start opennms, it will boot:




I saw several questions similar to this one and on the forums and I followed all the steps:

1) Check  ports to be open, firewall is offline:

[root@space-005056be0191 ~]# lsof -i :8980
COMMAND   PID   USER   FD   TYPE DEVICE SIZE NODE NAME
java    23233   root  494u  IPv6 267014       TCP *:8980 (LISTEN)
java    23233   root  495u  IPv6 267015       TCP localhost.localdomain:8980->localhost.localdomain:50023 (FIN_WAIT2)
java    23233   root  497u  IPv6 273540       TCP localhost.localdomain:50023->localhost.localdomain:8980 (CLOSE_WAIT)...
[root@space-005056be0191 ~]# netstat -tulpn | grep -e 8980 -e 8181
tcp        0      0 :::18980                    :::*                        LISTEN      23233/java
tcp        0      0 :::8980                     :::*                        LISTEN      23233/java
tcp        0      0 ::ffff:127.0.0.1:8181       :::*                        LISTEN      23233/java


2) openNMS is running
[root@space-005056be0191 ~]# opennms -v status
OpenNMS.Eventd         : running
OpenNMS.Trapd          : running
OpenNMS.Queued         : running
OpenNMS.Actiond        : running
OpenNMS.Notifd         : running
OpenNMS.Scriptd        : running
OpenNMS.Rtcd           : running
OpenNMS.Pollerd        : running
OpenNMS.Collectd       : running
OpenNMS.Threshd        : running
OpenNMS.Vacuumd        : running
OpenNMS.EventTranslator: running
OpenNMS.PassiveStatusd : running
OpenNMS.Statsd         : running
OpenNMS.Provisiond     : running
OpenNMS.Reportd        : running
OpenNMS.Alarmd         : running
OpenNMS.Ackd           : running
OpenNMS.Correlator     : running
OpenNMS.JettyServer    : running
OpenNMS.Syslogd        : running
opennms is running


3) There are no errors on /opt/opennms/logs/webapp

root@space-005056be0191 webapp]# ll
total 40
-rw-rw-r-- 1 root root 10079 May 24 23:41 jetty.log
-rw-rw-r-- 1 root root     0 May 24 23:41 rancid.log
-rw-rw-r-- 1 root root     0 May 24 23:41 tomcat-internal.log
-rw-rw-r-- 1 root root     0 May 24 23:41 webauth.log
-rw-rw-r-- 1 root root     0 May 24 23:41 web.log
-rw-rw-r-- 1 root root     0 May 24 23:41 web-map.log
-rw-rw-r-- 1 root root     0 May 24 23:41 web-rtc.log
[root@space-005056be0191 webapp]# grep ERROR *
There are only warnings like these:
2012-05-24 23:41:30,427 WARN  [Main] WebAppContext: Failed startup of context o.e.j.w.WebAppContext{/opennms,file:/opt/opennms/jetty-webapps/opennms/},/opt/opennms/jetty-webapps/opennms
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'reportSchedulerService' defined in class path resource [org/opennms/web/svclayer/applicationContext-svclayer.xml]: Cannot resolve reference to bean 'schedulerFactory' while setting bean property 'scheduler'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'schedulerFactory' defined in class path resource [org/opennms/web/svclayer/applicationContext-svclayer.xml]: Invocation of init method failed; nested exception is org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: No row exists in table QRTZ_LOCKS for lock named: TRIGGER_ACCESS [See nested exception: java.sql.SQLException: No row exists in table QRTZ_LOCKS for lock named: TRIGGER_ACCESS]]

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'schedulerFactory' defined in class path resource [org/opennms/web/svclayer/applicationContext-svclayer.xml]: Invocation of init method failed; nested exception is org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: No row exists in table QRTZ_LOCKS for lock named: TRIGGER_ACCESS [See nested exception: java.sql.SQLException: No row exists in table QRTZ_LOCKS for lock named: TRIGGER_ACCESS]]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
        ... 51 more
[root@space-005056be0191 daemon]# grep ERROR *
manager.log:2012-05-24 23:41:05,441 ERROR [Thread-9] Invoker: An error occurred invoking operation status on MBean OpenNMS:Name=Correlator: javax.management.RuntimeMBeanException: java.lang.NullPointerException
manager.log:2012-05-24 23:41:08,978 ERROR [Thread-11] Invoker: An error occurred invoking operation status on MBean OpenNMS:Name=Correlator: javax.management.RuntimeMBeanException: java.lang.NullPointerException
manager.log:2012-05-24 23:41:11,624 ERROR [Thread-13] Invoker: An error occurred invoking operation status on MBean OpenNMS:Name=Correlator: javax.management.RuntimeMBeanException: java.lang.NullPointerException
manager.log:2012-05-24 23:41:14,279 ERROR [Thread-15] Invoker: An error occurred invoking operation status on MBean OpenNMS:Name=Correlator: javax.management.RuntimeMBeanException: java.lang.NullPointerException
manager.log:2012-05-24 23:41:17,033 ERROR [Thread-18] Invoker: An error occurred invoking operation status on MBean OpenNMS:Name=Correlator: javax.management.RuntimeMBeanException: java.lang.NullPointerException
manager.log:2012-05-24 23:41:19,888 ERROR [Thread-21] Invoker: An error occurred invoking operation status on MBean OpenNMS:Name=Correlator: javax.management.RuntimeMBeanException: java.lang.NullPointerException
spring.log:2012-05-24 23:41:30,424 ERROR [Main] ContextLoader: Context initialization failed
[root@space-005056be0191 daemon]# pwd
/opt/opennms/logs/daemon



4) I read that many issues are caused by problems on the snmp-graph.properties and I removed the customized graphs just to make sure it would boot

Any suggestions? We have restarted all services, opennms, rebooted the machine several times, checked the files consistency with xmllint and everything is fine, as mentioned earlier no files were modified since original installation a month ago.

Thanks in advance
Reply | Threaded
Open this post in threaded view
|

Re: HTTP ERROR: 503 Problems accesing /opennms/index.jsp. Reason: service unavailable

LongIslandTechGuy
I have a similar error, here is the output from my /var/log/opennms/webapp/jetty.log file.

2013-02-16 19:28:50,149 WARN  [Main] JettyServer: adding context: /opennms -> /opt/opennms/jetty-webapps/opennms
2013-02-16 19:28:50,227 WARN  [Main] JettyServer: adding context: /opennms-remoting -> /opt/opennms/jetty-webapps/opennms-remoting
2013-02-16 19:28:50,228 WARN  [Main] JettyServer: adding context: / -> /opt/opennms/jetty-webapps/ROOT
2013-02-16 19:28:51,169 INFO  [Main] Server: jetty-7.5.1.v20110908
2013-02-16 19:29:16,583 WARN  [Main] WebAppContext: Failed startup of context o.e.j.w.WebAppContext{/opennms,file:/opt/opennms/jetty-webapps/opennms/},/opt/opennms/jetty-webapps/opennms
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'webAlarmRepository': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: org.opennms.netmgt.dao.MemoDao org.opennms.web.alarm.DaoWebAlarmRepository.m_memoDao; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [org.opennms.netmgt.dao.MemoDao] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:285)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1074)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)