OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Guenther Schreiner
Hello together,

 after the OpenNMS upgrade to 19.0.1 we have lost some parts of the GUI,
 i.e. the provisioning requisitions („cannot retrieve requisitions,… try again later“)
 and the resource graphs as well as the KSC reports.
 
 Reviewing the resource usage, I have increased the max. number of file handles
 to 20480 as well as the Java heap to -Xms2g -Xmx10g but in vain:
 there are still lots of "java.lang.OutOfMemoryError: unable to create new native thread“ messages…
 The nice fact is that the other operations like monitoring and BSM work well ?!

 Does anyone experience the same issues?

Regards,
 Guenther

BTW: the environment is
- CentOS release 6.9 (Final) (virtualized guest w/ 12 GB RAM)
- postgresql 9..4.8
- java 1.8.0_131




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Ronny Trommer-3
Hi Guenther,

Do you run on bare metal or a virtual machine? It seems to me, you run into an operation system resource problem which is not related to memory. Your box can’t spawn new threads. Especially in virtualised environments there are several reasons and policies which set a limit to the amount of threads which can be spawned. On plain bare metal setups this is an uncommon problem.

Can you check how many threads you have configured in the collectd-confiugration.xml and poller-configuration.xml? Otherwise how large is your environment with Nodes?

Greetings from Stuttgart

> On 24. Apr 2017, at 08:25, Guenther Schreiner <[hidden email]> wrote:
>
> Hello together,
>
> after the OpenNMS upgrade to 19.0.1 we have lost some parts of the GUI,
> i.e. the provisioning requisitions („cannot retrieve requisitions,… try again later“)
> and the resource graphs as well as the KSC reports.
>
> Reviewing the resource usage, I have increased the max. number of file handles
> to 20480 as well as the Java heap to -Xms2g -Xmx10g but in vain:
> there are still lots of "java.lang.OutOfMemoryError: unable to create new native thread“ messages…
> The nice fact is that the other operations like monitoring and BSM work well ?!
>
> Does anyone experience the same issues?
>
> Regards,
> Guenther
>
> BTW: the environment is
> - CentOS release 6.9 (Final) (virtualized guest w/ 12 GB RAM)
> - postgresql 9..4.8
> - java 1.8.0_131
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Norbert
Hi Ronny,
 
i'm fighting the same problem, which makes my test system completly unusable after several hours. No logon gui/console possible
It started also after upgrading from 18.0.2 to 19.0.1
 
I can confirm that this is a thread/spawn problem as the "out of memory" is missleading.
 
System
Ubuntu 16.04 LTS
VMWare VM (clone from productive, same setup)
4 cores, 16 GB
java version "1.8.0_111"
postgresql-9.3
 
Actual  ~ 4200 nodes
 
Thread settings (default)
collectd-configuration.xml              :   threads="50"
enlinkd-configuration.xml               :   threads="10"
poller-configuration.xml                :   threads="30"
snmp-interface-poller-configuration.xml :   threads="30"
threshd-configuration.xml               :   threads="5"
 
Process details:
/usr/lib/jvm/java-8-oracle/bin/java -Djava.endorsed.dirs=/usr/share/opennms/lib/endorsed -Dopennms.home=/usr/share/opennms -Xmx8196m -XX:+HeapDumpOnOutOfMemoryError -d64 -XX:+UseStringDeduplication -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseG1GC -Xloggc:/var/log/opennms/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=20M -Dcom.sun.management.jmxremote.port=18980 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dopennms.poller.server.serverHost=0.0.0.0 -Djava.rmi.server.hostname=10.3.72.246 -Dcom.sun.management.jmxremote.login.config=opennms -Dcom.sun.management.jmxremote.access.file=/usr/share/opennms/etc/jmxremote.access -DisThreadContextMapInheritable=true -Dgroovy.use.classvalue=true -XX:MaxMetaspaceSize=256m -Djava.io.tmpdir=/usr/share/opennms/data/tmp -jar /usr/share/opennms/lib/opennms_bootstrap.jar start
 
The productice VM  V18.0.2 runs well with these settings.
I talked to Jesse a few weeks ago and tried to get a heapdump, but this is also no possible.
 
Any suggestions?
 
Rgds Norbert
 
 
 
 
Gesendet: Montag, 24. April 2017 um 11:56 Uhr
Von: "Ronny Trommer" <[hidden email]>
An: "General OpenNMS Discussion" <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Hi Guenther,

Do you run on bare metal or a virtual machine? It seems to me, you run into an operation system resource problem which is not related to memory. Your box can’t spawn new threads. Especially in virtualised environments there are several reasons and policies which set a limit to the amount of threads which can be spawned. On plain bare metal setups this is an uncommon problem.

Can you check how many threads you have configured in the collectd-confiugration.xml and poller-configuration.xml? Otherwise how large is your environment with Nodes?

Greetings from Stuttgart

> On 24. Apr 2017, at 08:25, Guenther Schreiner <[hidden email]> wrote:
>
> Hello together,
>
> after the OpenNMS upgrade to 19.0.1 we have lost some parts of the GUI,
> i.e. the provisioning requisitions („cannot retrieve requisitions,… try again later“)
> and the resource graphs as well as the KSC reports.
>
> Reviewing the resource usage, I have increased the max. number of file handles
> to 20480 as well as the Java heap to -Xms2g -Xmx10g but in vain:
> there are still lots of "java.lang.OutOfMemoryError: unable to create new native thread“ messages…
> The nice fact is that the other operations like monitoring and BSM work well ?!
>
> Does anyone experience the same issues?
>
> Regards,
> Guenther
>
> BTW: the environment is
> - CentOS release 6.9 (Final) (virtualized guest w/ 12 GB RAM)
> - postgresql 9..4.8
> - java 1.8.0_131
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Norbert
In reply to this post by Ronny Trommer-3
Additional Info attached
 
Gesendet: Montag, 24. April 2017 um 11:56 Uhr
Von: "Ronny Trommer" <[hidden email]>
An: "General OpenNMS Discussion" <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Hi Guenther,

Do you run on bare metal or a virtual machine? It seems to me, you run into an operation system resource problem which is not related to memory. Your box can’t spawn new threads. Especially in virtualised environments there are several reasons and policies which set a limit to the amount of threads which can be spawned. On plain bare metal setups this is an uncommon problem.

Can you check how many threads you have configured in the collectd-confiugration.xml and poller-configuration.xml? Otherwise how large is your environment with Nodes?

Greetings from Stuttgart
 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

JVM_PEAK_THREAD_COUNT.JPG (257K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Guenther Schreiner
In reply to this post by Ronny Trommer-3
Hi Ronny,

 of course, there might be a problem with the system resources.

 But with 10 (ten!) threads for collectd and additional 10 threads for poller combined
 with less than 150 nodes this does not sound reasonable for even a small
 virtual environment (which was suitable for the previous releases ;-).

Best regards,
 Guenther

-----Ursprüngliche Nachricht-----
Von: Ronny Trommer [mailto:[hidden email]]
Gesendet: Montag, 24. April 2017 11:56
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Hi Guenther,

Do you run on bare metal or a virtual machine? It seems to me, you run into an operation system resource problem which is not related to memory. Your box can’t spawn new threads. Especially in virtualised environments there are several reasons and policies which set a limit to the amount of threads which can be spawned. On plain bare metal setups this is an uncommon problem.

Can you check how many threads you have configured in the collectd-confiugration.xml and poller-configuration.xml? Otherwise how large is your environment with Nodes?

Greetings from Stuttgart



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Ronny Trommer-3
Hi,

yes agree the number of threads in your configuration looks extremely descent. What really wondering is the graph provided by Norbert which seems to peek over 26.000 Threads.

Can we try to eliminate a probably candidate for a Thread leak by disabling the snap-interface poller daemon and enhanced linkd daemon.

Additionally a Thread dump instead of a heap dump would be more useful. You can create a thread dump by sending a signal to the Java process with:

kill -3 JAVA_PID

Most of the time I use this tool to analyse the thread dump which is available here: https://td.opennms.eu

Greetings



On 24. Apr 2017, at 12:46, Guenther Schreiner <[hidden email]> wrote:

Hi Ronny,

of course, there might be a problem with the system resources.

But with 10 (ten!) threads for collectd and additional 10 threads for poller combined
with less than 150 nodes this does not sound reasonable for even a small
virtual environment (which was suitable for the previous releases ;-).

Best regards,
Guenther

-----Ursprüngliche Nachricht-----
Von: Ronny Trommer [[hidden email]]
Gesendet: Montag, 24. April 2017 11:56
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Hi Guenther,

Do you run on bare metal or a virtual machine? It seems to me, you run into an operation system resource problem which is not related to memory. Your box can’t spawn new threads. Especially in virtualised environments there are several reasons and policies which set a limit to the amount of threads which can be spawned. On plain bare metal setups this is an uncommon problem.

Can you check how many threads you have configured in the collectd-confiugration.xml and poller-configuration.xml? Otherwise how large is your environment with Nodes?

Greetings from Stuttgart



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Ronny Trommer-3
snap-interface poller daemon

Sorry my autocorrection broke it, I mean the snmp-interface-poller-daemon


On 24. Apr 2017, at 13:46, Ronny Trommer <[hidden email]> wrote:

Hi,

yes agree the number of threads in your configuration looks extremely descent. What really wondering is the graph provided by Norbert which seems to peek over 26.000 Threads.

Can we try to eliminate a probably candidate for a Thread leak by disabling the snap-interface poller daemon and enhanced linkd daemon.

Additionally a Thread dump instead of a heap dump would be more useful. You can create a thread dump by sending a signal to the Java process with:

kill -3 JAVA_PID

Most of the time I use this tool to analyse the thread dump which is available here: https://td.opennms.eu

Greetings



On 24. Apr 2017, at 12:46, Guenther Schreiner <[hidden email]> wrote:

Hi Ronny,

of course, there might be a problem with the system resources.

But with 10 (ten!) threads for collectd and additional 10 threads for poller combined
with less than 150 nodes this does not sound reasonable for even a small
virtual environment (which was suitable for the previous releases ;-).

Best regards,
Guenther

-----Ursprüngliche Nachricht-----
Von: Ronny Trommer [[hidden email]]
Gesendet: Montag, 24. April 2017 11:56
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Hi Guenther,

Do you run on bare metal or a virtual machine? It seems to me, you run into an operation system resource problem which is not related to memory. Your box can’t spawn new threads. Especially in virtualised environments there are several reasons and policies which set a limit to the amount of threads which can be spawned. On plain bare metal setups this is an uncommon problem.

Can you check how many threads you have configured in the collectd-confiugration.xml and poller-configuration.xml? Otherwise how large is your environment with Nodes?

Greetings from Stuttgart



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Seth Leger-2
In reply to this post by Norbert
On these graphs, it does look like something is creating a lot of
threads unexpectedly. We need to catch your system right as this happens
and get a thread dump. It should be obvious what is causing the problem
in that thread dump.

You might set up a cron job that just runs jstack on OpenNMS every 5
minutes, you should be able to catch it then as long as jstack completes.

-- Seth


On 4/24/17 6:41 AM, Norbert Steinhoff wrote:

> Additional Info attached
>  
> *Gesendet:* Montag, 24. April 2017 um 11:56 Uhr
> *Von:* "Ronny Trommer" <[hidden email]>
> *An:* "General OpenNMS Discussion" <[hidden email]>
> *Betreff:* Re: [opennms-discuss] OpenNMS partially unusable after
> upgrade 18.0.3 -> 19.0.1
> Hi Guenther,
>
> Do you run on bare metal or a virtual machine? It seems to me, you run
> into an operation system resource problem which is not related to
> memory. Your box can’t spawn new threads. Especially in virtualised
> environments there are several reasons and policies which set a limit to
> the amount of threads which can be spawned. On plain bare metal setups
> this is an uncommon problem.
>
> Can you check how many threads you have configured in the
> collectd-confiugration.xml and poller-configuration.xml? Otherwise how
> large is your environment with Nodes?
>
> Greetings from Stuttgart
>  
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Norbert
In reply to this post by Ronny Trommer-3
Hi Ronny,
 
i see. :)
 
I dont think snmp-interface-poller-daemon is the root, because we monitor only 24 (missioncritical) interfaces
and in our case it runs every 20 seconds... (interval="20000")
opennms=# select snmppoll,count(*) from snmpinterface group by 1;
 snmppoll | count
----------+--------
 N        | 126034
 P        |     24
 
 
So first step i'll try  >> disabled EnhancedLinkd
 
Will report if this was a hit ....
 
Rgds
Norbert
 
Actually running:
opennms-test:/usr/share/opennms/logs# opennms -v status
OpenNMS.Eventd         : running
OpenNMS.Trapd          : running
OpenNMS.Queued         : running
OpenNMS.Actiond        : running
OpenNMS.Notifd         : running
OpenNMS.Scriptd        : running
OpenNMS.Rtcd           : running
OpenNMS.Pollerd        : running
OpenNMS.PollerBackEnd  : running
OpenNMS.SnmpPoller     : running
OpenNMS.Ticketer       : running
OpenNMS.Collectd       : running
OpenNMS.Discovery      : running
OpenNMS.Vacuumd        : running
OpenNMS.EventTranslator: running
OpenNMS.PassiveStatusd : running
OpenNMS.Statsd         : running
OpenNMS.Provisiond     : running
OpenNMS.Reportd        : running
OpenNMS.Bsmd           : running
OpenNMS.Alarmd         : running
OpenNMS.Ackd           : running
OpenNMS.JettyServer    : running
opennms is running
 
 
 
Gesendet: Montag, 24. April 2017 um 13:47 Uhr
Von: "Ronny Trommer" <[hidden email]>
An: "General OpenNMS Discussion" <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
snap-interface poller daemon
 
Sorry my autocorrection broke it, I mean the snmp-interface-poller-daemon
 
 
On 24. Apr 2017, at 13:46, Ronny Trommer <[hidden email]> wrote:
 
Hi,
 
yes agree the number of threads in your configuration looks extremely descent. What really wondering is the graph provided by Norbert which seems to peek over 26.000 Threads.
 
Can we try to eliminate a probably candidate for a Thread leak by disabling the snap-interface poller daemon and enhanced linkd daemon.
 
Additionally a Thread dump instead of a heap dump would be more useful. You can create a thread dump by sending a signal to the Java process with:
 
kill -3 JAVA_PID
 
Most of the time I use this tool to analyse the thread dump which is available here: https://td.opennms.eu
 
Greetings
 
 
 
On 24. Apr 2017, at 12:46, Guenther Schreiner <[hidden email]> wrote:
 
Hi Ronny,

of course, there might be a problem with the system resources.

But with 10 (ten!) threads for collectd and additional 10 threads for poller combined
with less than 150 nodes this does not sound reasonable for even a small
virtual environment (which was suitable for the previous releases ;-).

Best regards,
Guenther

-----Ursprüngliche Nachricht-----
Von: Ronny Trommer [[hidden email]]
Gesendet: Montag, 24. April 2017 11:56
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Hi Guenther,

Do you run on bare metal or a virtual machine? It seems to me, you run into an operation system resource problem which is not related to memory. Your box can’t spawn new threads. Especially in virtualised environments there are several reasons and policies which set a limit to the amount of threads which can be spawned. On plain bare metal setups this is an uncommon problem.

Can you check how many threads you have configured in the collectd-confiugration.xml and poller-configuration.xml? Otherwise how large is your environment with Nodes?

Greetings from Stuttgart



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-discuss mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Norbert
In reply to this post by Seth Leger-2
Hi Seth,
 
i created a cronjob every 2 min with
 
/usr/bin/jstack $(ps -ef | grep java|grep -v grep| awk '{ print $2}')  > /mnt/sdc2/td_$(date +%Y%m%d_%H%M%S)
 
and it collects.
 
Ronny suggested to disable enhancedlinkd, but i think while fetching with  jstack i will go back to original setup with
enhancedlinkd enabled.
 
Will come back if i have more info and hopefully some thread dumps.
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 16:45 Uhr
Von: "Seth Leger" <[hidden email]>
An: "General OpenNMS Discussion" <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
On these graphs, it does look like something is creating a lot of
threads unexpectedly. We need to catch your system right as this happens
and get a thread dump. It should be obvious what is causing the problem
in that thread dump.

You might set up a cron job that just runs jstack on OpenNMS every 5
minutes, you should be able to catch it then as long as jstack completes.

-- Seth


On 4/24/17 6:41 AM, Norbert Steinhoff wrote:
> Additional Info attached
>
> *Gesendet:* Montag, 24. April 2017 um 11:56 Uhr
> *Von:* "Ronny Trommer" <[hidden email]>
> *An:* "General OpenNMS Discussion" <[hidden email]>
> *Betreff:* Re: [opennms-discuss] OpenNMS partially unusable after
> upgrade 18.0.3 -> 19.0.1
> Hi Guenther,
>
> Do you run on bare metal or a virtual machine? It seems to me, you run
> into an operation system resource problem which is not related to
> memory. Your box can’t spawn new threads. Especially in virtualised
> environments there are several reasons and policies which set a limit to
> the amount of threads which can be spawned. On plain bare metal setups
> this is an uncommon problem.
>
> Can you check how many threads you have configured in the
> collectd-confiugration.xml and poller-configuration.xml? Otherwise how
> large is your environment with Nodes?
>
> Greetings from Stuttgart
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

David Hustace
In reply to this post by Ronny Trommer-3

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer <[hidden email]> wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (210 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Guenther Schreiner
In reply to this post by Ronny Trommer-3
Hi Ronny,

 strange - even with only few services, we’re hitting the resource limits:

        [root@server opennms]# opennms -v status
        OpenNMS.Eventd         : running
        OpenNMS.Trapd          : running
        OpenNMS.Queued         : running
        OpenNMS.Rtcd           : running
        OpenNMS.Pollerd        : running
        OpenNMS.PollerBackEnd  : running
        OpenNMS.EventTranslator: running
        OpenNMS.Provisiond     : running
        OpenNMS.JettyServer    : running

 Regards,
 Guenther
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Norbert
In reply to this post by David Hustace
Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david <[hidden email]>
An: "DiscussionList List" <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer <[hidden email]> wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Jesse White-3
Norbert;

Can you compress all of those and attach them to this issue:
     https://issues.opennms.org/browse/NMS-9233

I suspect this is a similar case.

Best,
Jesse

On 04/24/2017 01:56 PM, Norbert Steinhoff wrote:
Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david [hidden email]
An: "DiscussionList List" [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer [hidden email] wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Norbert
Hi Jesse,
 
Done
 
Gesendet: Montag, 24. April 2017 um 23:10 Uhr
Von: "Jesse White" <[hidden email]>
An: [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Norbert;

Can you compress all of those and attach them to this issue:
     https://issues.opennms.org/browse/NMS-9233

I suspect this is a similar case.

Best,
Jesse
 
On 04/24/2017 01:56 PM, Norbert Steinhoff wrote:
Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david [hidden email]
An: "DiscussionList List" [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer [hidden email] wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
 
 
 
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
 
 
 
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-discuss mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Jesse White-3
Thanks for the thread dumps - this confirms my suspicions.

The JIRA issue contains more details and the patched .jars that should address the problem:
    https://issues.opennms.org/browse/NMS-9233

-Jesse

On 04/24/2017 02:17 PM, Norbert Steinhoff wrote:
Hi Jesse,
 
Done
 
Gesendet: Montag, 24. April 2017 um 23:10 Uhr
Von: "Jesse White" [hidden email]
An: [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Norbert;

Can you compress all of those and attach them to this issue:
     https://issues.opennms.org/browse/NMS-9233

I suspect this is a similar case.

Best,
Jesse
 
On 04/24/2017 01:56 PM, Norbert Steinhoff wrote:
Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david [hidden email]
An: "DiscussionList List" [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer [hidden email] wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
 
 
 
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
 
 
 
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-discuss mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Ronny Trommer-3
In reply to this post by Norbert
You can try to upload them to 


Expires on 2017-05-31

Thanks

On 24. Apr 2017, at 22:56, Norbert Steinhoff <[hidden email]> wrote:

Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david <[hidden email]>
An: "DiscussionList List" <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer <[hidden email]> wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Ronny Trommer-3
I see, you just added them to NMS-9233, I had done the same, forget then about the upload link.


On 25. Apr 2017, at 10:21, Ronny Trommer <[hidden email]> wrote:

You can try to upload them to 


Expires on 2017-05-31

Thanks

On 24. Apr 2017, at 22:56, Norbert Steinhoff <[hidden email]> wrote:

Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david <[hidden email]>
An: "DiscussionList List" <[hidden email]>
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer <[hidden email]> wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Norbert
In reply to this post by Jesse White-3
Hi Jesse,
 
since applying that patches from NMS-9233  -> no more crashes .
Works as expected for me.
 
Nice Weekend
Norbert
 
Gesendet: Dienstag, 25. April 2017 um 00:29 Uhr
Von: "Jesse White" <[hidden email]>
An: [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Thanks for the thread dumps - this confirms my suspicions.

The JIRA issue contains more details and the patched .jars that should address the problem:
    https://issues.opennms.org/browse/NMS-9233

-Jesse

On 04/24/2017 02:17 PM, Norbert Steinhoff wrote:
Hi Jesse,
 
Done
 
Gesendet: Montag, 24. April 2017 um 23:10 Uhr
Von: "Jesse White" [hidden email]
An: [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Norbert;

Can you compress all of those and attach them to this issue:
     https://issues.opennms.org/browse/NMS-9233

I suspect this is a similar case.

Best,
Jesse
 
On 04/24/2017 01:56 PM, Norbert Steinhoff wrote:
Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david [hidden email]
An: "DiscussionList List" [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer [hidden email] wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
 
 
 
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
 
 
 
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-discuss mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-discuss
 
 
 
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
 
 
 
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-discuss mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

Jesse White-3
Happy to hear. We'll get that patched merged in for the next release.

Thanks again for providing us with the details to isolate it.

Best,
Jesse

On 04/28/2017 10:19 AM, Norbert Steinhoff wrote:
Hi Jesse,
 
since applying that patches from NMS-9233  -> no more crashes .
Works as expected for me.
 
Nice Weekend
Norbert
 
Gesendet: Dienstag, 25. April 2017 um 00:29 Uhr
Von: "Jesse White" [hidden email]
An: [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Thanks for the thread dumps - this confirms my suspicions.

The JIRA issue contains more details and the patched .jars that should address the problem:
    https://issues.opennms.org/browse/NMS-9233

-Jesse

On 04/24/2017 02:17 PM, Norbert Steinhoff wrote:
Hi Jesse,
 
Done
 
Gesendet: Montag, 24. April 2017 um 23:10 Uhr
Von: "Jesse White" [hidden email]
An: [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1
Norbert;

Can you compress all of those and attach them to this issue:
     https://issues.opennms.org/browse/NMS-9233

I suspect this is a similar case.

Best,
Jesse
 
On 04/24/2017 01:56 PM, Norbert Steinhoff wrote:
Hi,
 
crash started  22:08 and ONMS was unresponsive 22:38
I got some jstack dumps . Where should i upload them and which of the following do you need ?
 
-rw-r--r-- 1 root root 394K Apr 24 22:02 td_20170424_220201
-rw-r--r-- 1 root root 371K Apr 24 22:04 td_20170424_220401
-rw-r--r-- 1 root root 389K Apr 24 22:06 td_20170424_220601
-rw-r--r-- 1 root root 2.7M Apr 24 22:08 td_20170424_220801
-rw-r--r-- 1 root root 2.8M Apr 24 22:10 td_20170424_221001
-rw-r--r-- 1 root root 4.2M Apr 24 22:12 td_20170424_221201
-rw-r--r-- 1 root root 5.7M Apr 24 22:14 td_20170424_221401
-rw-r--r-- 1 root root 5.9M Apr 24 22:16 td_20170424_221601
-rw-r--r-- 1 root root 8.7M Apr 24 22:18 td_20170424_221801
-rw-r--r-- 1 root root 8.8M Apr 24 22:20 td_20170424_222001
-rw-r--r-- 1 root root  11M Apr 24 22:22 td_20170424_222201
-rw-r--r-- 1 root root  12M Apr 24 22:24 td_20170424_222401
-rw-r--r-- 1 root root  12M Apr 24 22:26 td_20170424_222601
-rw-r--r-- 1 root root  15M Apr 24 22:28 td_20170424_222801
-rw-r--r-- 1 root root  15M Apr 24 22:30 td_20170424_223001
-rw-r--r-- 1 root root  17M Apr 24 22:32 td_20170424_223201
-rw-r--r-- 1 root root  18M Apr 24 22:34 td_20170424_223401
-rw-r--r-- 1 root root  18M Apr 24 22:36 td_20170424_223601
-rw-r--r-- 1 root root    0 Apr 24 22:38 td_20170424_223801
-rw-r--r-- 1 root root    0 Apr 24 22:40 td_20170424_224001
-rw-r--r-- 1 root root    0 Apr 24 22:42 td_20170424_224201
 
 
Rgds
Norbert
 
 
Gesendet: Montag, 24. April 2017 um 22:23 Uhr
Von: david [hidden email]
An: "DiscussionList List" [hidden email]
Betreff: Re: [opennms-discuss] OpenNMS partially unusable after upgrade 18.0.3 -> 19.0.1

> On Apr 24, 2017, at 7:46 AM, Ronny Trommer [hidden email] wrote:
>
> kill -3 JAVA_PID

echo "doing 3 thread dumps..." >&2;for i in {1..3}; do echo "dump: $i" >&2; sudo jstack -l `cat /var/log/opennms/opennms.pid `; echo "sleeping 60..." >&2; sleep 60; echo "thread dumps complete." >&2; done > /tmp/opennms-thread.dump

Getting a few thread dumps a minute apart is often helpful for comparisons.


David


David Hustace
The OpenNMS Group, Inc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
 
 
 
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
 
 
 
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-discuss mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-discuss
 
 
 
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
 
 
 
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-discuss mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
12