OutOfMemory Exception

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

OutOfMemory Exception

Florian Laufer (DARC e. V.)

Hello everybody,

 

we’re getting an OutOfMemory Exception, there are several entries in the output.log like:

 

Exception in thread "pool-20128-thread-1" java.lang.OutOfMemoryError: unable to create new native thread

 

How can we setup Heap Size etc. right for 80 Nodes?`

 

Actual workaround is restarting the vm every night.

 

opennms.conf:

 

JAVA_HEAP_SIZE=1536

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+UseG1GC"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -Xloggc:/var/log/opennms/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=20M"

ADDITIONAL_MANAGER_OPTIONS="-Xms"$JAVA_HEAP_SIZE"m

 

The vm has 3,5 GB of RAM, more is of course possible.

 

What could be the right setup?

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OutOfMemory Exception

Roskens, Ronald-2

The "OutOfMemoryError" might be misleading. The full error does not indicate that you are running into a java memory issue, ie: heap. It’s the second part "unable to create new native thread" that's the issue, and that indicates there is an issue at the OS layer.

 

Have you looked at top/vmstat/systat/etc. to see how the OS is performing?

 

You probably want to bump the VM memory to 6GB, and bump the JVM heap to 2GB. If the VM has 1 vCPU, you might want to increase it to 4 vCPU or more. Again, top/vmstat/etc. would be a good starting point to look at how the OS is performing.

 

Ron

 

From: Florian Laufer (DARC e. V.) [mailto:[hidden email]]
Sent: Tuesday, March 07, 2017 8:38 AM
To: General OpenNMS Discussion
Subject: [opennms-discuss] OutOfMemory Exception

 

Hello everybody,

 

we’re getting an OutOfMemory Exception, there are several entries in the output.log like:

 

Exception in thread "pool-20128-thread-1" java.lang.OutOfMemoryError: unable to create new native thread

 

How can we setup Heap Size etc. right for 80 Nodes?`

 

Actual workaround is restarting the vm every night.

 

opennms.conf:

 

JAVA_HEAP_SIZE=1536

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+UseG1GC"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -Xloggc:/var/log/opennms/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=20M"

ADDITIONAL_MANAGER_OPTIONS="-Xms"$JAVA_HEAP_SIZE"m

 

The vm has 3,5 GB of RAM, more is of course possible.

 

What could be the right setup?

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OutOfMemory Exception

David Hustace
In reply to this post by Florian Laufer (DARC e. V.)

> On Mar 7, 2017, at 9:38 AM, Florian Laufer (DARC e. V.) <[hidden email]> wrote:
>
> Hello everybody,
>  
> we’re getting an OutOfMemory Exception, there are several entries in the output.log like:

What version of OpenNMS are you running?
Is this a fresh install?
What Java version? (32-bit?, 64-bit?)
What OS? (32-bit?, 64-bit?)

David


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OutOfMemory Exception

Florian Laufer (DARC e. V.)
Hi David,

we're running 1.8.0 60 64-Bit JDK on a OpenSuse 42.2 Leap 64-Bit with OpenNMS 18.0.3. It's an old install.

--
Mit freundlichen Grüßen / Best regards
--------------------------------------------------------
Florian Laufer
-Team-IT-
Deutscher Amateur-Radio-Club e. V.
Lindenallee 4
34225 Baunatal
Tel.: 0561 94988-92
Fax: 0561 94988-50
E-Mail: [hidden email]
Web: http://www.darc.de
--------------------------------------------------------
Vereinsregister Kassel, VR 1314
Geschäftsführer: Jens Hergert
--------------------------------------------------------

-----Ursprüngliche Nachricht-----
Von: David Hustace [mailto:[hidden email]]
Gesendet: Dienstag, 7. März 2017 19:09
An: DiscussionList List <[hidden email]>
Betreff: Re: [opennms-discuss] OutOfMemory Exception


> On Mar 7, 2017, at 9:38 AM, Florian Laufer (DARC e. V.) <[hidden email]> wrote:
>
> Hello everybody,
>
> we’re getting an OutOfMemory Exception, there are several entries in the output.log like:

What version of OpenNMS are you running?
Is this a fresh install?
What Java version? (32-bit?, 64-bit?)
What OS? (32-bit?, 64-bit?)

David


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned dictionary content that is easy and intuitive to access. Sign up for an account today to start using our lexical data to power your apps and projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
[Logo_FunkTagKassel]<https://www.darc.de/nachrichten/veranstaltungen/#c35435>

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!<https://www.darc.de/nachrichten/veranstaltungen/#c35435>

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.
------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OutOfMemory Exception

Florian Laufer (DARC e. V.)
In reply to this post by Roskens, Ronald-2

Hi Ron,

 

vmstat / top shows around 400MB free RAM, i’ll follow your advices.

 

Now we have 4 vCPU, 6GB of VM Memory and 2GB of heap size. We’ll see the effect tomorrow.

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Von: Roskens, Ronald [mailto:[hidden email]]
Gesendet: Dienstag, 7. März 2017 16:35
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OutOfMemory Exception

 

The "OutOfMemoryError" might be misleading. The full error does not indicate that you are running into a java memory issue, ie: heap. It’s the second part "unable to create new native thread" that's the issue, and that indicates there is an issue at the OS layer.

 

Have you looked at top/vmstat/systat/etc. to see how the OS is performing?

 

You probably want to bump the VM memory to 6GB, and bump the JVM heap to 2GB. If the VM has 1 vCPU, you might want to increase it to 4 vCPU or more. Again, top/vmstat/etc. would be a good starting point to look at how the OS is performing.

 

Ron

 

From: Florian Laufer (DARC e. V.) [[hidden email]]
Sent: Tuesday, March 07, 2017 8:38 AM
To: General OpenNMS Discussion
Subject: [opennms-discuss] OutOfMemory Exception

 

Hello everybody,

 

we’re getting an OutOfMemory Exception, there are several entries in the output.log like:

 

Exception in thread "pool-20128-thread-1" java.lang.OutOfMemoryError: unable to create new native thread

 

How can we setup Heap Size etc. right for 80 Nodes?`

 

Actual workaround is restarting the vm every night.

 

opennms.conf:

 

JAVA_HEAP_SIZE=1536

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+UseG1GC"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -Xloggc:/var/log/opennms/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=20M"

ADDITIONAL_MANAGER_OPTIONS="-Xms"$JAVA_HEAP_SIZE"m

 

The vm has 3,5 GB of RAM, more is of course possible.

 

What could be the right setup?

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OutOfMemory Exception

Florian Laufer (DARC e. V.)

So, OpenNMS hung again.

 

TOP:

 

top - 08:52:50 up 1 day, 16:04,  1 user,  load average: 0,03, 0,07, 0,08

Tasks: 245 total,   1 running, 233 sleeping,   0 stopped,  11 zombie

%Cpu(s):  0,1 us,  0,2 sy,  0,0 ni, 99,8 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st

KiB Mem:   6110452 total,  3184304 used,  2926148 free,   168184 buffers

KiB Swap:  1051644 total,        0 used,  1051644 free.  1317136 cached Mem

 

Vmstat:

 

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

1  0      0 2926264 168188 1317104    0    0     2    90    2   33  5  6 89  0  0

 

Errors in output.log:

 

hundreds of entries like:

 

org.opennms.netmgt.events.api.EventProxyException: Unknown exception while sending event: java.lang.OutOfMemoryError: unable to create new native thread

 

So, what can i do?

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Von: Florian Laufer (DARC e. V.) [mailto:[hidden email]]
Gesendet: Mittwoch, 8. März 2017 16:48
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OutOfMemory Exception

 

Hi Ron,

 

vmstat / top shows around 400MB free RAM, i’ll follow your advices.

 

Now we have 4 vCPU, 6GB of VM Memory and 2GB of heap size. We’ll see the effect tomorrow.

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Von: Roskens, Ronald [[hidden email]]
Gesendet: Dienstag, 7. März 2017 16:35
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OutOfMemory Exception

 

The "OutOfMemoryError" might be misleading. The full error does not indicate that you are running into a java memory issue, ie: heap. It’s the second part "unable to create new native thread" that's the issue, and that indicates there is an issue at the OS layer.

 

Have you looked at top/vmstat/systat/etc. to see how the OS is performing?

 

You probably want to bump the VM memory to 6GB, and bump the JVM heap to 2GB. If the VM has 1 vCPU, you might want to increase it to 4 vCPU or more. Again, top/vmstat/etc. would be a good starting point to look at how the OS is performing.

 

Ron

 

From: Florian Laufer (DARC e. V.) [[hidden email]]
Sent: Tuesday, March 07, 2017 8:38 AM
To: General OpenNMS Discussion
Subject: [opennms-discuss] OutOfMemory Exception

 

Hello everybody,

 

we’re getting an OutOfMemory Exception, there are several entries in the output.log like:

 

Exception in thread "pool-20128-thread-1" java.lang.OutOfMemoryError: unable to create new native thread

 

How can we setup Heap Size etc. right for 80 Nodes?`

 

Actual workaround is restarting the vm every night.

 

opennms.conf:

 

JAVA_HEAP_SIZE=1536

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+UseG1GC"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -Xloggc:/var/log/opennms/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=20M"

ADDITIONAL_MANAGER_OPTIONS="-Xms"$JAVA_HEAP_SIZE"m

 

The vm has 3,5 GB of RAM, more is of course possible.

 

What could be the right setup?

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OutOfMemory Exception

Dino Yancey
Increase the number of threads available to the OpenNMS user.  See /etc/security/limits.conf (or the equivalent on Suse)

On Fri, Mar 10, 2017 at 1:55 AM, Florian Laufer (DARC e. V.) <[hidden email]> wrote:

So, OpenNMS hung again.

 

TOP:

 

top - 08:52:50 up 1 day, 16:04,  1 user,  load average: 0,03, 0,07, 0,08

Tasks: 245 total,   1 running, 233 sleeping,   0 stopped,  11 zombie

%Cpu(s):  0,1 us,  0,2 sy,  0,0 ni, 99,8 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st

KiB Mem:   6110452 total,  3184304 used,  2926148 free,   168184 buffers

KiB Swap:  1051644 total,        0 used,  1051644 free.  1317136 cached Mem

 

Vmstat:

 

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

1  0      0 2926264 168188 1317104    0    0     2    90    2   33  5  6 89  0  0

 

Errors in output.log:

 

hundreds of entries like:

 

org.opennms.netmgt.events.api.EventProxyException: Unknown exception while sending event: java.lang.OutOfMemoryError: unable to create new native thread

 

So, what can i do?

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Von: Florian Laufer (DARC e. V.) [mailto:[hidden email]]
Gesendet: Mittwoch, 8. März 2017 16:48


An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OutOfMemory Exception

 

Hi Ron,

 

vmstat / top shows around 400MB free RAM, i’ll follow your advices.

 

Now we have 4 vCPU, 6GB of VM Memory and 2GB of heap size. We’ll see the effect tomorrow.

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Von: Roskens, Ronald [[hidden email]]
Gesendet: Dienstag, 7. März 2017 16:35
An: General OpenNMS Discussion <[hidden email]>
Betreff: Re: [opennms-discuss] OutOfMemory Exception

 

The "OutOfMemoryError" might be misleading. The full error does not indicate that you are running into a java memory issue, ie: heap. It’s the second part "unable to create new native thread" that's the issue, and that indicates there is an issue at the OS layer.

 

Have you looked at top/vmstat/systat/etc. to see how the OS is performing?

 

You probably want to bump the VM memory to 6GB, and bump the JVM heap to 2GB. If the VM has 1 vCPU, you might want to increase it to 4 vCPU or more. Again, top/vmstat/etc. would be a good starting point to look at how the OS is performing.

 

Ron

 

From: Florian Laufer (DARC e. V.) [[hidden email]]
Sent: Tuesday, March 07, 2017 8:38 AM
To: General OpenNMS Discussion
Subject: [opennms-discuss] OutOfMemory Exception

 

Hello everybody,

 

we’re getting an OutOfMemory Exception, there are several entries in the output.log like:

 

Exception in thread "pool-20128-thread-1" java.lang.OutOfMemoryError: unable to create new native thread

 

How can we setup Heap Size etc. right for 80 Nodes?`

 

Actual workaround is restarting the vm every night.

 

opennms.conf:

 

JAVA_HEAP_SIZE=1536

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -XX:+UseG1GC"

ADDITIONAL_MANAGER_OPTIONS="${ADDITIONAL_MANAGER_OPTIONS} -Xloggc:/var/log/opennms/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=20M"

ADDITIONAL_MANAGER_OPTIONS="-Xms"$JAVA_HEAP_SIZE"m

 

The vm has 3,5 GB of RAM, more is of course possible.

 

What could be the right setup?

 

--

Mit freundlichen Grüßen / Best regards

--------------------------------------------------------

Florian Laufer

-Team-IT-

Deutscher Amateur-Radio-Club e. V.

Lindenallee 4

34225 Baunatal

Tel.: 0561 94988-92

Fax: 0561 94988-50

E-Mail: [hidden email]

Web: http://www.darc.de

--------------------------------------------------------

Vereinsregister Kassel, VR 1314

Geschäftsführer: Jens Hergert

--------------------------------------------------------

 

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.

Logo_FunkTagKassel

2. FUNK.TAG: Erleben Sie die Faszination Technik auf der Funk-Erlebnismesse am 8. April 2017 in Kassel!

Diese E-Mail enthält vertrauliche und / oder rechtlich geschützte Informationen und diese sind ausschließlich DARC intern zu verwenden. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das Kopieren sowie die Weitergabe oder Veröffentlichung dieser Mail nebst Anhängen, auch auszugsweise, ist nur mit vorheriger schriftlich unterzeichneter Genehmigung des Verfassers gestattet.

This e-mail may contain confidential and / or privileged information and are only to be used DARC internally. If you are not the intended recipient or have received this e-mail in error, please notify the sender immediately and destroy this e-mail. Unauthorized copying, unauthorized disclosure or publication of this mail with its contents and attachments, even in part, is permitted only with prior written signed permission of the author.


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Loading...