Issue polling services

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue polling services

TAN Lists
Hey all,

I have found a strange issue today that i can not get my head around.

One of the OpenNMS servers we have is running fine, one isn't - they operate as unique nodes, each OpenNMS server doesnt know about the other, but monitor a test node for me to play around with.

Server A can see all the services on the server, server B refuses to detect HTTP, HTTPS, FTP and MySQL

No firewalls, no communication issues, telnet test from the OpenNMS server works fine. HTTP returning 200ok

I've diff'd the files and copied some of them over to see if that solves it, updated both of them to the new 1.10.4 version and still no dice :(

When i click rescan, it finishes within 2 seconds and doesn't detect any HTTP services etc - if i remove all services and rescan, same thing. If i delete the node, same thing. If i delete the foreign source and recreate, same thing, if i create a new provisioning group, same thing

I'm outa ideas - this is a stock standard install and was working until a few days ago, no changes made

Only relevant thing i can find in the logs is:

[root@monitoring daemon]# grep -i "2012-07-18 13:53:59,960 INFO" provisiond.log
2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] AsyncBasicDetectorMinaImpl$2: Connection exception occurred java.net.SocketException: Invalid argument for service HTTP, retrying attempt 1
2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] AsyncBasicDetectorMinaImpl$2: Service HTTP detected false
2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] IpInterfaceScan$1: Attempted to detect service HTTP on address xxx.xxx.xxx.xxx: false
[root@monitoring daemon]#

Invalid argument for service HTTP, retrying attempt 1 <-- This appears for all services it wont detect :(

Any ideas guys, im pulling my hair out (needed a haircut anyway, so good timing!)

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

TAN Lists
Hey all,

Just googled another part of the error and found this
http://sourceforge.net/mailarchive/forum.php?thread_name=4FC51838.8010801%40opennms.org&forum_name=opennms-discuss

I have attached my logs in a paste bin here - http://pastebin.com/qK5q2Bp7

Logging was already on Debug

Any thoughts guys?

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

TAN Lists
In reply to this post by TAN Lists
Just looking back through some logs - i think this might be related to an upgrade to the 1.10.3 version from an earlier one.

I did a fresh install of 1.10.3 on another server and the issue is not apparent.

On my development server i jumped from 1.8.16 to 1.10.3 and then to 1.10.4 today and the issue is not apparent.

The 2 servers where i do see this have jumped from version to version each time.

Let me know if i can be of help, i would like to get this solved for anyone else out there, but may end up removing OpenNMS completely and reinstalling

All servers run CentOS 5.8 x64 with OpenNMS installed from repo stable

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

TAN Lists
Was anyone able to replicate / solve this??


On 18 July 2012 16:32, TAN Lists <[hidden email]> wrote:
Just looking back through some logs - i think this might be related to an upgrade to the 1.10.3 version from an earlier one.

I did a fresh install of 1.10.3 on another server and the issue is not apparent.

On my development server i jumped from 1.8.16 to 1.10.3 and then to 1.10.4 today and the issue is not apparent.

The 2 servers where i do see this have jumped from version to version each time.

Let me know if i can be of help, i would like to get this solved for anyone else out there, but may end up removing OpenNMS completely and reinstalling

All servers run CentOS 5.8 x64 with OpenNMS installed from repo stable


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

vincent333
In reply to this post by TAN Lists
Hi , nice to meet you all. i am very glad that can join this forum. this is my first time to reply thread. hope we can exchangee some useful information among us. enjoy your day ^^




--------------------------------------------------------------------------------------------------------

Largest Express bus Ticket Website (Singapore & Malaysia)
Introduction of Express Bus service from Singapore to and fro Malaysia
The most comprehensive tourists guide in Singapore!
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

TAN Lists
In reply to this post by TAN Lists
I ended up formatting the servers (needed these new servers online) and i couldn't find a solution

Sorry to anyone that ends up here from Google

On 18 July 2012 15:04, TAN Lists <[hidden email]> wrote:
Hey all,

I have found a strange issue today that i can not get my head around.

One of the OpenNMS servers we have is running fine, one isn't - they operate as unique nodes, each OpenNMS server doesnt know about the other, but monitor a test node for me to play around with.

Server A can see all the services on the server, server B refuses to detect HTTP, HTTPS, FTP and MySQL

No firewalls, no communication issues, telnet test from the OpenNMS server works fine. HTTP returning 200ok

I've diff'd the files and copied some of them over to see if that solves it, updated both of them to the new 1.10.4 version and still no dice :(

When i click rescan, it finishes within 2 seconds and doesn't detect any HTTP services etc - if i remove all services and rescan, same thing. If i delete the node, same thing. If i delete the foreign source and recreate, same thing, if i create a new provisioning group, same thing

I'm outa ideas - this is a stock standard install and was working until a few days ago, no changes made

Only relevant thing i can find in the logs is:

[root@monitoring daemon]# grep -i "2012-07-18 13:53:59,960 INFO" provisiond.log
2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] AsyncBasicDetectorMinaImpl$2: Connection exception occurred java.net.SocketException: Invalid argument for service HTTP, retrying attempt 1
2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] AsyncBasicDetectorMinaImpl$2: Service HTTP detected false
2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] IpInterfaceScan$1: Attempted to detect service HTTP on address xxx.xxx.xxx.xxx: false
[root@monitoring daemon]#

Invalid argument for service HTTP, retrying attempt 1 <-- This appears for all services it wont detect :(

Any ideas guys, im pulling my hair out (needed a haircut anyway, so good timing!)


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

John Blake

What does your config look like for the service?




Inactive hide details for TAN Lists ---07/30/2012 01:05:42 AM---I ended up formatting the servers (needed these new servers onlTAN Lists ---07/30/2012 01:05:42 AM---I ended up formatting the servers (needed these new servers online) and i couldn't find a solution

From: TAN Lists <[hidden email]>
To: General OpenNMS Discussion <[hidden email]>,
Date: 07/30/2012 01:05 AM
Subject: Re: [opennms-discuss] Issue polling services





I ended up formatting the servers (needed these new servers online) and i couldn't find a solution

Sorry to anyone that ends up here from Google

On 18 July 2012 15:04, TAN Lists <[hidden email]> wrote:
    Hey all,

    I have found a strange issue today that i can not get my head around.

    One of the OpenNMS servers we have is running fine, one isn't - they operate as unique nodes, each OpenNMS server doesnt know about the other, but monitor a test node for me to play around with.

    Server A can see all the services on the server, server B refuses to detect HTTP, HTTPS, FTP and MySQL

    No firewalls, no communication issues, telnet test from the OpenNMS server works fine. HTTP returning 200ok

    I've diff'd the files and copied some of them over to see if that solves it, updated both of them to the new 1.10.4 version and still no dice :(

    When i click rescan, it finishes within 2 seconds and doesn't detect any HTTP services etc - if i remove all services and rescan, same thing. If i delete the node, same thing. If i delete the foreign source and recreate, same thing, if i create a new provisioning group, same thing

    I'm outa ideas - this is a stock standard install and was working until a few days ago, no changes made

    Only relevant thing i can find in the logs is:

    [root@monitoring daemon]# grep -i "2012-07-18 13:53:59,960 INFO" provisiond.log
    2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] AsyncBasicDetectorMinaImpl$2: Connection exception occurred java.net.SocketException: Invalid argument for service HTTP, retrying attempt 1
    2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] AsyncBasicDetectorMinaImpl$2: Service HTTP detected false
    2012-07-18 13:53:59,960 INFO  [Provisiond-TaskScheduler] IpInterfaceScan$1: Attempted to detect service HTTP on address xxx.xxx.xxx.xxx: false
    [root@monitoring daemon]#

    Invalid argument for service HTTP, retrying attempt 1 <-- This appears for all services it wont detect :(

    Any ideas guys, im pulling my hair out (needed a haircut anyway, so good timing!)
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats.
http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

TAN Lists
In reply to this post by TAN Lists
I stand corrected

New install of OpenNMS 1.10.4 on CentOS 5.8 x64 still has the issue

The OpenNMS install wont detect the simple detector services

Java Version: 1.7.0_05 Oracle Corporation
Java Virtual Machine: 23.1-b03 Oracle Corporation

I know i don't pay for support here, but can someone from OpenNMS acknowledge if this is an issue or if i am doing something wrong?

This is a present issue stopping it from working and i believe is a bug

I've deployed OpenNMS 20 or so times now and I've never ever seen it do this :/
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

Cyrille Bollu
Same behaviour here; I've opened an issue on JIRA http://issues.opennms.org/browse/NMS-5469

BR,

Cyrille

On Tue, Jul 31, 2012 at 9:20 AM, TAN Lists <[hidden email]> wrote:
I stand corrected

New install of OpenNMS 1.10.4 on CentOS 5.8 x64 still has the issue

The OpenNMS install wont detect the simple detector services

Java Version: 1.7.0_05 Oracle Corporation
Java Virtual Machine: 23.1-b03 Oracle Corporation

I know i don't pay for support here, but can someone from OpenNMS acknowledge if this is an issue or if i am doing something wrong?

This is a present issue stopping it from working and i believe is a bug

I've deployed OpenNMS 20 or so times now and I've never ever seen it do this :/
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

TAN Lists
Good to know someone else is seeing it

I was seriously starting to pull my hair out - I just sent the network techs rogue on the issue thinking something was filtering the connection, ill go call them off now :)

Be good to see some input from the OpenNMS team here, this is a pretty serious issue IMHO

One thing i did just see that is weirder

We have 4 of these servers 3 in Sydney and 1 in Brisbane, all deployed at the same time and installed via cluster SSH so all commands were identical

Sydney servers are having the issue, but the BNE one can detect services fine. But servers in Sydney cant detect HTTP on external services
I would be willing to say that it was the network / filtering if it wasnt so weird (deployed same time, same version, same commands, no firewalls, SELinux is off permanantly, routing is fine and can telnet test the services from the effected server) There is something weird in the service triggering the issue

If i knew Java at all, i would go rip[ the thing to shreds, but java looses me at hello

On 31 July 2012 18:28, Cyrille Bollu <[hidden email]> wrote:
Same behaviour here; I've opened an issue on JIRA http://issues.opennms.org/browse/NMS-5469

BR,

Cyrille

On Tue, Jul 31, 2012 at 9:20 AM, TAN Lists <[hidden email]> wrote:
I stand corrected

New install of OpenNMS 1.10.4 on CentOS 5.8 x64 still has the issue

The OpenNMS install wont detect the simple detector services

Java Version: 1.7.0_05 Oracle Corporation
Java Virtual Machine: 23.1-b03 Oracle Corporation

I know i don't pay for support here, but can someone from OpenNMS acknowledge if this is an issue or if i am doing something wrong?

This is a present issue stopping it from working and i believe is a bug

I've deployed OpenNMS 20 or so times now and I've never ever seen it do this :/
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Issue polling services

Cyrille Bollu
Hello,

I've done some debugging on this subject, and updated http://issues.opennms.org/browse/NMS-5469 accordingly.

Also, I'm wondering if it's a platform specific issue. My 2 openNMS boxes both run on Debian amd64. Is this also your platform?

BR,

Cyrille

Le 31 juil. 2012 à 10:40, TAN Lists a écrit :

Good to know someone else is seeing it

I was seriously starting to pull my hair out - I just sent the network techs rogue on the issue thinking something was filtering the connection, ill go call them off now :)

Be good to see some input from the OpenNMS team here, this is a pretty serious issue IMHO

One thing i did just see that is weirder

We have 4 of these servers 3 in Sydney and 1 in Brisbane, all deployed at the same time and installed via cluster SSH so all commands were identical

Sydney servers are having the issue, but the BNE one can detect services fine. But servers in Sydney cant detect HTTP on external services
I would be willing to say that it was the network / filtering if it wasnt so weird (deployed same time, same version, same commands, no firewalls, SELinux is off permanantly, routing is fine and can telnet test the services from the effected server) There is something weird in the service triggering the issue

If i knew Java at all, i would go rip[ the thing to shreds, but java looses me at hello

On 31 July 2012 18:28, Cyrille Bollu <[hidden email]> wrote:
Same behaviour here; I've opened an issue on JIRA http://issues.opennms.org/browse/NMS-5469

BR,

Cyrille

On Tue, Jul 31, 2012 at 9:20 AM, TAN Lists <[hidden email]> wrote:
I stand corrected

New install of OpenNMS 1.10.4 on CentOS 5.8 x64 still has the issue

The OpenNMS install wont detect the simple detector services

Java Version: 1.7.0_05 Oracle Corporation
Java Virtual Machine: 23.1-b03 Oracle Corporation

I know i don't pay for support here, but can someone from OpenNMS acknowledge if this is an issue or if i am doing something wrong?

This is a present issue stopping it from working and i believe is a bug

I've deployed OpenNMS 20 or so times now and I've never ever seen it do this :/
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss