OpenNMS and syslog relaying

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

OpenNMS and syslog relaying

Jimisola Laursen
Administrator
Hi!
 
I have some questions regarding the SysLog functionality in OpenNMS.
 
The problem is that OpenNMS syslog stores wrong node and interface when relaying syslog messages via syslog-ng
 
Background:
One central server using syslog-ng receives syslog messages from various machines (Solaris, Linux, Suse, RH etc).
Both syslog-ng and OpenNMS shall receive syslog messages and OpenNMS shall, of course, receive/have syslog messages with correct facility, priority, source host etc.

The standard syslog (which exists on some clients/nodes/interfaces) cannot specify destination port (e.g. 10514) and we want to limit the network traffic, open ports etc. We therefore send everything to the central syslog-ng server on standard port 514 and then relay it to OpenNMS on port 10514 (OpenNMS is installed on the central syslog-ng server).

This usage has been discussed in forums online as well, see e.g. http://www.linuxquestions.org/questions/linux-server-73/syslog.conf-alternate-port-644664/
 
By relaying the node and interface is set to the central syslog-ng instead of the original syslog message sender.
 
Example:
 
Prereqs:
 
host1/1.1.1.1 = central syslog-ng and OpenNMS server
host2/1.1.1.5 = host sending the actual syslog message
 
In OpenNMS:
 
Node: host1.lan.net
Interface: 1.1.1.1
 
Log Message:
 
An OpenNMS Event has been received as a Syslog Message
Message: host2 sshd[18929]: [ID 800047 auth.info] Did not receive ident string from 1.1.1.2.
 
Description:
 
The interface 1.1.1.1 generated a Syslog Message.
Node ID: 199
Host: host1.lan.net
Interface: 1.1.1.1
Message: seldmip_om sshd[18929]: [ID 800047 auth.info] Did not receive ident string from 1.1.1.2.
Process:
PID: 0
 
Would it be possible to add optional relay-aware functionality to OpenNMS Syslog support so that it can parse the Message (here: "seldmip_om sshd[18929]: [ID 800047 auth.info] Did not receive ident string from 1.1.1.2.") to set Node and Interface as well as Description part correct?
 
In the above example this would mean
 
Log Message:
 
Node: host2.lan.net
Interface: 1.1.1.5
 
Description:
 
The interface 1.1.1.5 generated a Syslog Message (via relay from host1.lan.net/1.1.1.1)
Node ID: 200
Host: host2.lan.net
Interface: 1.1.1.5
Message: seldmip_om sshd[18929]: [ID 800047 auth.info] Did not receive ident string from 1.1.1.2.
Process:
PID: 0
 
Best Regards,
Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
I received feedback from Johan Edstrom on this issue before I posted to this mailing list:

"You need to define you regexp correctly for this. Look in the syslog config."

So, I continued...

syslogd-configuration.xml:

    <configuration
            syslog-port="10514"
            new-suspect-on-message="true"
            forwarding-regexp="^.*\s(19|20)\d\d([-/.])(0[1-9]|1[012])\2(0[1-9]|[12][0-9]|3[01])(\s+)(\S+)(\s)(\S.+)"
            matching-group-host="6"
            matching-group-message="8"
            />

This regexp seems to match, e.g. "2008-07-13 some-none-whitespace some-none-whitespace".

However, our syslog data has a different date format (of course).
So, I wrote a new regexp (leaving out the handling of PID for now) and changed match-group-*:

           

    <configuration
            syslog-port="10514"
            new-suspect-on-message="true"
            forwarding-regexp="^\s*(Jan|Feb|Mar|Apr|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s(0[1-9]|[12][0-9]|3[01])\s(\d\d:\d\d:\d\d)\s(\S+)\s(\S+):(.+)$"
            matching-group-host="4"
            matching-group-message="6"

            />


However, in OpenNMS the syslog-ng server still shows as the Node and not the host in syslog message!

I tested the regexp outside of OpenNMS (using beanshell) and it works there.
I'm unable to verify (at least to my knowledge) that the regexp works in OpenNMS.
In logs/daemon/syslogd.log I see the following:

2008-07-14 14:48:01,824 DEBUG [Syslog Event Receiver[10514]] SyslogReceiver: Wating on a datagram to arrive
2008-07-14 14:48:01,824 DEBUG [Thread-3551] Syslogd: In the make part of UdpReceivedSyslog org.opennms.netmgt.syslogd.ConvertToEvent@a813ccc
2008-07-14 14:48:01,825 DEBUG [Thread-3551] Syslogd: Message : unx10 cron[12767]: (root) CMD (/usr/local/shell/count_httpd.sh)

2008-07-14 14:48:01,825 DEBUG [Thread-3551] Syslogd: Pattern : ^\s*(Jan|Feb|Mar|Apr|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s(0[1-9]|[12][0-9]|3[01])\s(\d\d:\d\d:\d\d)\s
(\S+)\s(\S+):(.+)$
2008-07-14 14:48:01,825 DEBUG [Thread-3551] Syslogd: Host group: 4
2008-07-14 14:48:01,825 DEBUG [Thread-3551] Syslogd: Message group: 6
2008-07-14 14:48:01,825 DEBUG [Thread-3551] Syslogd: Attempting substring match for text of a Syslogd event to :CRISCO
2008-07-14 14:48:01,825 DEBUG [Thread-3551] Syslogd: Attempting regex match for text of a Syslogd event to :.*fancyd: .*failed for user (\S+) on ((pts\/\d+)|(
tty\d+)).*
2008-07-14 14:48:01,825 DEBUG [Thread-3551] SyslogConnection: Sending received packet to the queue
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor: Processing a syslog to event dispatchorg.opennms.netmgt.syslogd.ConvertToEvent@
a813ccc
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor: Event {
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   uuid  = <not-set>
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   uei   = uei.opennms.org/syslogd/cron/Info
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   src   = syslogd
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   iface = 10.129.236.131
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   time  = Monday, July 14, 2008 12:48:01 PM GMT
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   Msg   = unx10 cron[12767]: (root) CMD (/usr/local/shell/count_httpd.sh)

2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   Dst   = logndisplay
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   parms {
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:     (syslogmessage, unx10 cron[12767]: (root) CMD (/usr/local/shell/count_h
ttpd.sh))
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:     (severity, Info)
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:     (timestamp, Jul 14 14:48:01)
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:     (process, )
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:     (service, cron)
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:     (processid, 0)
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor:   }
2008-07-14 14:48:01,825 DEBUG [Syslog Event Processor[10514]] SyslogProcessor: }

1) Can someone confirm from the debug information that my regexp works? If not, I suggest additional debug output such as whether the regexp
matches or not (appplies to <ueiList><ueiMatch> as well).

2) iface = "10.129.236.131" is the interface from which the syslog message is relayed (i.e. the syslog-ng server) and not the originating syslog message machine. I suspect that this is why Node still is not correct in OpenNMS -> Events -> Detail. Again, is there something wrong in my regexp or have misunderstood something about syslogd-configuration.xml?

3) Process and PID under Events -> Detail -> Description are empty. How do I set these? Are there match-group-process and match-group-PID available? It would be useful to be able to set this information as well.

I'll gladly submit my working regexp when it is done. Then whether the user should have to switch manually in syslogd-configuration.xml between the regexp depending on datetime format or if both should be active is for the development team to decide.

Regards,
Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jeff Gehlbach
On Jul 16, 2008, at 6:52 AM, Jimisola Laursen wrote:

> I received feedback from Johan Edstrom on this issue before I posted  
> to this
> mailing list:
>
> "You need to define you regexp correctly for this. Look in the syslog
> config."

Johan is right, configuring Syslogd's forwarding-regexp is the key to  
syslog happiness.  You might try the much simpler one described here:

http://www.opennms.org/index.php/Syslogd#Configuration

I find that this one works very well in a syslog-ng forwarding  
environment.  Good choice on that point, by the way.

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator

Jeff Gehlbach wrote
On Jul 16, 2008, at 6:52 AM, Jimisola Laursen wrote:

> I received feedback from Johan Edstrom on this issue before I posted  
> to this
> mailing list:
>
> "You need to define you regexp correctly for this. Look in the syslog
> config."

Johan is right, configuring Syslogd's forwarding-regexp is the key to  
syslog happiness.  You might try the much simpler one described here:

http://www.opennms.org/index.php/Syslogd#Configuration

I find that this one works very well in a syslog-ng forwarding  
environment.  Good choice on that point, by the way.

-jeff
Well, the forwarding-regexp event/message solution is not really what we wanted.
I had hoped that once the main syslog event was in OpenNMS the notification would be a piece of cake.
Sadly, this means that we have to edit the xml file each time we find a new syslog description than we need a notification for.

I'll see what we'll do. We have cacti and Groundworks running as well. Perhaps they can handle it in a smoother way.

Thank you for your reply.

Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
In reply to this post by Jeff Gehlbach
Jeff,

My three questions regarding setting correct host/iface, process and PID and verifying the matches remains unanswered I believe. Or did I miss out on something?

Regards,
Jimisola

Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jeff Gehlbach
In reply to this post by Jimisola Laursen
On Jul 16, 2008, at 3:21 PM, Jimisola Laursen wrote:
> Well, the forwarding-regexp event/message solution is not really  
> what we
> wanted.

That's how you make specific event types from syslog messages that  
match what you want.  If you write your regexes carefully, you can use  
the support in Syslogd for back-references to populate parameters in  
the resulting events with values that you can key on using the  
parameter binding feature of notifications.

> I had hoped that once the main syslog event was in OpenNMS the  
> notification
> would be a piece of cake.

Once an event is in the system, making a notification for it IS a  
piece of cake.  You're just making assumptions about the best places  
to do filtering that are entirely understandable for a newcomer to the  
OpenNMS community.

> Sadly, this means that we have to edit the xml file each time we  
> find a new
> syslog description than we need a notification for.

I'm sorry if you find editing a file to be a hardship.  It's the  
mechanism we've got at present.  You're welcome to open an enhancement  
bug describing the way you think it should work.

> I'll see what we'll do. We have cacti and Groundworks running as well.
> Perhaps they can handle it in a smoother way.

Last time I checked, Cacti was just a tool for gathering and graphing  
time-series data and could not receive SNMP traps.

Using Groundwork Open Source (sic), it appears you'll have to set up  
syslog-ng to write the messages you want to a log file, manually  
configure a Nagios plugin to check each host's log file against a list  
of regexes in a file that you'll have to edit -- http://is.gd/Vce.  
That puts you in the same boat, except you'll also be doing disk I/O  
to get at the syslog messages.

Use whatever works for you.  OpenNMS can do what you want with no  
extra software apart from syslog-ng as an aggregation layer.  You  
could look into IBM Tivoli Netcool OMNIbus with a syslog probe, but  
you'll have to learn the Netcool probe rules quasi-language because  
you still have to edit a file, in this case a probe rules file.  
That's after you pay $USD 100,000+ up front for licenses and 20% of  
that annually for maintenance.

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
Jeff Gehlbach wrote
On Jul 16, 2008, at 3:21 PM, Jimisola Laursen wrote:
> Well, the forwarding-regexp event/message solution is not really  
> what we
> wanted.

That's how you make specific event types from syslog messages that  
match what you want.  If you write your regexes carefully, you can use  
the support in Syslogd for back-references to populate parameters in  
the resulting events with values that you can key on using the  
parameter binding feature of notifications.
Ok. What you explained in the last sentence is exactly what I'm looking for.
May I ask where I find an example that illustrates this?
I've looked around (mostly at http://www.opennms.org/index.php/Syslogd), but I must have missed it.

I'm interested in knowing the available parametersand how to set them using back-reference (I'll be glad to extend the Syslogd page once it's clear to me).
Host, process and PID are the most important parameters but I also have some custom parameters that I would like to set.

My forward-regexp still seems valid since our syslog uses a different datetime format than the one used per default in syslogd-configuration.xml. Correct?

In my particular case I need to be able to keep track of status (OK or NOT OK in the syslog message) for the combination of the host from which the syslog message originated and an id in the syslog message itself. Sounds like this should be doable if I just can create a custom event and set parameters accordingly.

Jeff Gehlbach wrote
> I had hoped that once the main syslog event was in OpenNMS the  
> notification
> would be a piece of cake.

Once an event is in the system, making a notification for it IS a  
piece of cake.  You're just making assumptions about the best places  
to do filtering that are entirely understandable for a newcomer to the  
OpenNMS community.


> Sadly, this means that we have to edit the xml file each time we  
> find a new
> syslog description than we need a notification for.

I'm sorry if you find editing a file to be a hardship.  It's the  
mechanism we've got at present.  You're welcome to open an enhancement  
bug describing the way you think it should work.
Hardship? I've edited many files in my life... :)
Jeff, don't get me wrong. I'm not trying to be critical about OpenNMs or make faulty assumptions.

I'm simply talking about it from a user perspective.
Ideally, for me as a user, everything would be configurable from the web interface.
I define a forward-regexp and can verify that it works directly in the web interface (even see the match groups), add "<ueiMatch>" etc.

Jeff Gehlbach wrote
Last time I checked, Cacti was just a tool for gathering and graphing  
time-series data and could not receive SNMP traps.
That's correct. I'm still quite new to this area. I was told to use Cacti, OpenNMS or Groundworks whatever solved our needs and after a quick look I feel for OpenNMS.

Jeff Gehlbach wrote
Using Groundwork Open Source (sic), it appears you'll have to set up  
syslog-ng to write the messages you want to a log file, manually  
configure a Nagios plugin to check each host's log file against a list  
of regexes in a file that you'll have to edit -- http://is.gd/Vce.  
That puts you in the same boat, except you'll also be doing disk I/O  
to get at the syslog messages.

Use whatever works for you.  OpenNMS can do what you want with no  
extra software apart from syslog-ng as an aggregation layer.
Well, it appears as if what I thought would be a quite easy task has grown a bit more complex than I first expected. With that said, according to your information on Groundworks OpenNMS is still the best of the two.

Thanks for you quick and informative response.

Regards,
Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Byron Anderson
In reply to this post by Jeff Gehlbach
> That's how you make specific event types from syslog messages that  
> match what you want.  If you write your regexes carefully, you can use

> the support in Syslogd for back-references to populate parameters in  
> the resulting events with values that you can key on using the  
> parameter binding feature of notifications.

Jeff, just wanted to let you know that I really like this approach and
am going to have to give it a try!

As far as the rest of this post is concerned, I wanted to point out that
we have a need for the exact same type of syslog notification thing and
we currently use a version of OpenNMS that doesn't support syslog.  We
solved the problem by using syslog-ng combined with Swatch.  Syslog-ng
captures the logs and sends them through Swatch which watches for
specific things based on regexe filters.  When Swatch catches something
it executes send-event.pl to trip an event that has been created for the
log.  This is a longer method than what Jeff has described but it worked
out well for us before this new syslog feature was available in OpenNMS.

No matter what tool you use (we used to use HP OpenView as well) you
will have to configure something for each new log that you want to be
notified about.  

I have also used Nagios/Groundworks and personally find OpenNMS to be a
lot better.  I have been using OpenNMS since it was in beta and the fact
that it was built as an enterprise grade monitoring solution from the
ground up is obvious when compared to some other systems.  The community
support is awesome and the support plan that the OpenNMS Group offers is
probably some of the best support I have seen in the industry.

If you find another tool that works better for you then I would say go
for it but I would seriously give OpenNMS a chance.  It takes a few
months to get a good feel for it.  The OpenNMS Group also offers a
wonderful training class that you maybe should consider if you choose to
go with it as your solution.  

-------------------------------------------
Byron Anderson
Network Operations Center Manager
EasyStreet Online Services
www.easystreet.com
 



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jeff Gehlbach
In reply to this post by Jimisola Laursen
On Jul 16, 2008, at 4:45 PM, Jimisola Laursen wrote:
>> If you write your regexes carefully, you can use
>> the support in Syslogd for back-references to populate parameters in
>> the resulting events with values that you can key on using the
>> parameter binding feature of notifications.
>>
>
> Ok. What you explained in the last sentence is exactly what I'm  
> looking for.
> May I ask where I find an example that illustrates this?

The default syslogd-configuration.xml has such an example, albeit a  
theoretical one.

https://opennms.svn.sourceforge.net/svnroot/opennms/opennms/branches/1.6-testing/opennms-daemon/src/main/filtered/etc/syslogd-configuration.xml

> I've looked around (mostly at http://www.opennms.org/index.php/Syslogd)
> , but
> I must have missed it.

The second <ueiMatch> in the quote block under Configuration in that  
page shows a regex match that includes subexpressions, but it does not  
explain how the back-references get turned into event parms.  The  
default config file does briefly show this.

> I'm interested in knowing the available parametersand how to set  
> them using
> back-reference (I'll be glad to extend the Syslogd page once it's  
> clear to
> me).

One quick way to see all the parameters of an event is to go look at  
the event in the web UI.  They'll be presented in a nice tabular format.

> Host, process and PID are the most important parameters but I also  
> have some
> custom parameters that I would like to set.

Anything that you can put into a subexpression of your regex can be an  
event parameter.

> My forward-regexp still seems valid since our syslog uses a different
> datetime format than the one used per default in syslogd-
> configuration.xml.
> Correct?

If your forwarding-regexp is working and is causing the resulting  
events to appear on the correct node, then it's at least somewhat valid.

> In my particular case I need to be able to keep track of status (OK  
> or NOT
> OK in the syslog message) for the combination of the host from which  
> the
> syslog message originated and an id in the syslog message itself.  
> Sounds
> like this should be doable if I just can create a custom event and set
> parameters accordingly.

If you annotate the definitions for your syslog events with an alarm-
data element (see OPENNMS_HOME/etc/events/Veraz.events.xml for a good  
example), then you can craft a reduction key that creates one alarm  
for whatever tuple you desire.  That's where the real power of back-
reference-sourced event parameters becomes apparent.

> Ideally, for me as a user, everything would be configurable from the  
> web
> interface.
> I define a forward-regexp and can verify that it works directly in  
> the web
> interface (even see the match groups), add "<ueiMatch>" etc.

Ideally for me as a project maintainer and booster, the same would be  
true.  Unfortunately we're not there yet, but we're taking steps in  
that direction.

> Well, it appears as if what I thought would be a quite easy task has  
> grown a
> bit more complex than I first expected.

Yeah, network management is hard.

> With that said, according to your
> information on Groundworks OpenNMS is still the best of the two.

Careful, I'm demonstrably biased :)

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Johan Edstrom
In reply to this post by Byron Anderson
I've extended the use-case for a former contract I worked on to also
if anyone is interested, generate Alarms from 'droves' of Syslog -  
info....

http://www.opennms.org/index.php/Syslogd_Automations



On Jul 16, 2008, at 3:00 PM, Byron Anderson wrote:

>> That's how you make specific event types from syslog messages that
>> match what you want.  If you write your regexes carefully, you can  
>> use
>
>> the support in Syslogd for back-references to populate parameters in
>> the resulting events with values that you can key on using the
>> parameter binding feature of notifications.
>
> Jeff, just wanted to let you know that I really like this approach and
> am going to have to give it a try!
>
> As far as the rest of this post is concerned, I wanted to point out  
> that
> we have a need for the exact same type of syslog notification thing  
> and
> we currently use a version of OpenNMS that doesn't support syslog.  We
> solved the problem by using syslog-ng combined with Swatch.  Syslog-ng
> captures the logs and sends them through Swatch which watches for
> specific things based on regexe filters.  When Swatch catches  
> something
> it executes send-event.pl to trip an event that has been created for  
> the
> log.  This is a longer method than what Jeff has described but it  
> worked
> out well for us before this new syslog feature was available in  
> OpenNMS.
>
> No matter what tool you use (we used to use HP OpenView as well) you
> will have to configure something for each new log that you want to be
> notified about.
>
> I have also used Nagios/Groundworks and personally find OpenNMS to  
> be a
> lot better.  I have been using OpenNMS since it was in beta and the  
> fact
> that it was built as an enterprise grade monitoring solution from the
> ground up is obvious when compared to some other systems.  The  
> community
> support is awesome and the support plan that the OpenNMS Group  
> offers is
> probably some of the best support I have seen in the industry.
>
> If you find another tool that works better for you then I would say go
> for it but I would seriously give OpenNMS a chance.  It takes a few
> months to get a good feel for it.  The OpenNMS Group also offers a
> wonderful training class that you maybe should consider if you  
> choose to
> go with it as your solution.
>
> -------------------------------------------
> Byron Anderson
> Network Operations Center Manager
> EasyStreet Online Services
> www.easystreet.com
>
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's  
> challenge
> Build the coolest Linux based applications with Moblin SDK & win  
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in  
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom  
> of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss

Johan Edstrom
[hidden email]




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Antw: Re: OpenNMS and syslog relaying

Michael Seibold
A good point to start with syslog in opennms is to use collected syslogs and check which entries occured during the last year or so (depending on how far back you have syslogs ;-). This way you will have most entries to work on "in the first round" and don't have to change event configuration etc. too often.

Here is an example to check for cisco log entries in a syslog-ng logfile named syslog_info. For these lines Parameter 9 starts with "%" and ends with ":", this delimits the cisco syslog message-type. This script runs on solaris with nawk, maybe you have to change it a little bit on other machines as they don't have nawk, but probably awk is working the same way there.


nawk '$9 ~ "^%.*:$" { typ[$9]++; }
      END {
          for (var in typ)
              { printf "%8d\t%s\n",typ[var],var;}
        }' syslog_info \
| sort -n -k1



What you will get is a list of the following type

No. of message-type
Entries
       1 %DTP-5-TRUNKPORTON:
       1 %DUAL-5-NBRCHANGE:
       1 %SNMP-5-COLDSTART:
       1 %SYS-5-RESTART:
       2 %DTP-SP-5-NONTRUNKPORTON:
       2 %DTP-SP-5-TRUNKPORTON:
       2 %STANDBY-6-STATECHANGE:
       2 %SW_MATM-4-MACFLAP_NOTIF:
       3 %SPANTREE-5-TOPOTRAP:
       4 %CLEAR-5-COUNTERS:
      14 %SEC-6-IPACCESSLOGNP:
      17 %SYS-5-PRIV_AUTH_PASS:
      23 %PM_SCP-SP-3-LCP_FW_ABLC:
      32 %SEC-6-IPACCESSLOGS:
      67 %LINK-5-CHANGED:
      82 %PM-4-ERR_RECOVER:
      95 %PM-4-ERR_DISABLE:
     145 %SYS-5-CONFIG_I:
     339 %SEC-6-IPACCESSLOGDP:
     597 %LINEPROTO-SP-5-UPDOWN:
     621 %LINK-SP-3-UPDOWN:
    1091 %IPNAT-6-NAT_CREATED:
    1133 %IPNAT-6-NAT_DELETED:
    9596 %SEC-6-IPACCESSLOGRL:
   42477 %LINK-3-UPDOWN:
   44328 %LINEPROTO-5-UPDOWN:
  110466 %SEC-6-IPACCESSLOGP:

With this information it is much easier to start working on the event-configuration in opennms.


Michael


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
In reply to this post by Jeff Gehlbach
Hi Jeff,

I've been reading up after you pinpointing me to the right sources and I must say that I understand OpenNMS much better now and I like what I see. It's powerful in deed!

After spending some time I came up with some questions (and one or two suggestions):

 1. own events: Where do I define my own Syslogd sub-events? I would like to use a new file and avoid events/Syslogd.events.xml to ease future upgrades of OpenNMS. Are all events/*.events.xml files read or do I have to add my own event file somewhere.

 2. %parms: How do I set %parm[process], %parm[processid] etc for later usage in an event creation definition?

 3. custom event fields: I think I might need to define custom event fields in order for me to be able to create notifications based on such a field later (or am I missing something). How do I define such an event field (e.g. phonenumber)?

 4. Where do I find information on the structure of the different .xml documents suchs as *.event.xml. It would mean the world if the .xml files had a DTD (with comments in DTD).

Suggestions:

 1. The basic idea of Syslogd handling and possibilities in OpenNMS, flow of events and event creation per se (usage of back-reference etc) should be added/updated on http://www.opennms.org/index.php/Syslogd? I got I much clearer picture of what OpenNMS is able to do after your readup suggestion than just reading the web page and there is a risk that others might miss out on the power as well.

 2. More informative examples or at least link to them from the Syslogd web page.

Regards,
Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
In reply to this post by Johan Edstrom
Just realized that the %parm[process], %parm[processid] etc must be named capturing groups.
Correct me if I'm wrong.

Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jeff Gehlbach
In reply to this post by Jimisola Laursen
On Jul 17, 2008, at 5:02 AM, Jimisola Laursen wrote:
> 1. own events: Where do I define my own Syslogd sub-events? I would  
> like to
> use a new file and avoid events/Syslogd.events.xml to ease future  
> upgrades
> of OpenNMS. Are all events/*.events.xml files read or do I have to  
> add my
> own event file somewhere.

You can and should create your own file.  The convention is to place  
these in etc/events.  These get included via <event-file> elements in  
etc/eventconf.xml.

> 2. %parms: How do I set %parm[process], %parm[processid] etc for later
> usage in an event creation definition?

Currently there's no way to set named parameters from back-references  
in syslogd-configuration.xml.  At present each back-reference gets  
turned into a parameter whose name is the number of the back-
reference.  So for the following regex match:

(?s)(((Some)|(No))body) set up (.*?) (.*)$

The following event parameters would be created when we receive  
"Somebody set up us the bomb":

0: Somebody set up us the bomb
1: Somebody
2: Some
3: Some
4: (blank)
5: us
6: the bomb

> 3. custom event fields: I think I might need to define custom event  
> fields
> in order for me to be able to create notifications based on such a  
> field
> later (or am I missing something). How do I define such an event  
> field (e.g.
> phonenumber)?

There are no custom event fields, but you can name an event parameter  
anything you like.  The kind of event enrichment that you're  
describing here is really what Automations are for.  See "Automations"  
on the wiki.

> 4. Where do I find information on the structure of the different .xml
> documents suchs as *.event.xml. It would mean the world if the .xml  
> files
> had a DTD (with comments in DTD).

Actually we use XSDs rather than DTDs.  Documentation on the XSDs is  
linked from the wiki; see http://www.opennms.org/documentation/java-xsddocs-stable/

> 1. The basic idea of Syslogd handling and possibilities in OpenNMS,  
> flow of
> events and event creation per se (usage of back-reference etc)  
> should be
> added/updated on http://www.opennms.org/index.php/Syslogd?

It's a wiki, which means anybody can edit...

> I got I much clearer picture of what OpenNMS is able to do after  
> your readup suggestion
> than just reading the web page and there is a risk that others might  
> miss
> out on the power as well.

Syslogd has been in OpenNMS for a while, but it's been only in the  
past year that Johan enhanced it to create custom events and I to do  
regex matches and back-reference extraction into event parameters.  
The docs haven't caught up yet, that's all :)

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
Jeff Gehlbach wrote
On Jul 17, 2008, at 5:02 AM, Jimisola Laursen wrote:
You can and should create your own file.  The convention is to place  
these in etc/events.  These get included via <event-file> elements in  
etc/eventconf.xml.
Ok.

Jeff Gehlbach wrote
> 3. custom event fields: I think I might need to define custom event  
> fields
> in order for me to be able to create notifications based on such a  
> field
> later (or am I missing something). How do I define such an event  
> field (e.g.
> phonenumber)?

There are no custom event fields, but you can name an event parameter  
anything you like.  The kind of event enrichment that you're  
describing here is really what Automations are for.  See "Automations"  
on the wiki.
I read the wiki page "Automations", but I don't understand how that helps me.
The syslog message has information that I via a fine-grained regexp can access using back-references (e.g. process and process id). I now thought I needed to save this information as a parameter in the event so that I can access it later. I would think that naming an event parameter would be the same as a custom event parameter. Appearently, it's not. I'm really not following and I believe it's because I don't have the full knowledge about how OpenNMS works.

What I want to accomplish is "simple":

 1. extract information from syslog message
 (2. create a custom event based on the message (using ueiMatch))
 3. associate certain syslog values to the event for later access

Jeff Gehlbach wrote
Actually we use XSDs rather than DTDs.  Documentation on the XSDs is  
linked from the wiki; see http://www.opennms.org/documentation/java-xsddocs-stable/
Even better. Thank you.

Jeff Gehlbach wrote
It's a wiki, which means anybody can edit...
I'll most likely do that as soon as I have the full picture myself :)

Jeff Gehlbach wrote
Syslogd has been in OpenNMS for a while, but it's been only in the  
past year that Johan enhanced it to create custom events and I to do  
regex matches and back-reference extraction into event parameters.  
The docs haven't caught up yet, that's all :)
Ok. Well, it's useful functionality that might draw people (like myself) to OpenNMS.

Regards,
Jimisola

PS. The automation page has "To access this column data, specify the column using the format '${<column name>}'". Looking at the code it looks as if this is wrong it should be '%{column name}' (note % instead of $). DS.
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jeff Gehlbach
On Jul 17, 2008, at 5:27 PM, Jimisola Laursen wrote:
> I read the wiki page "Automations", but I don't understand how that  
> helps
> me.

You indicated you want to add stuff like a phone number.  In an  
automation, you can dive into an external database via JDBC and find  
stuff like contact numbers if it's not already populated in the node's  
asset information.  On the other hand, if it is in an asset field  
already, you can get at it since OpenNMS 1.5.94 in a notification  
definition by saying e.g. %asset[supportphone]%.

> The syslog message has information that I via a fine-grained regexp  
> can
> access using back-references (e.g. process and process id). I now  
> thought I
> needed to save this information as a parameter in the event so that  
> I can
> access it later.

This happens implicitly and with no extra configuration needed.  The  
resulting parms are named "1", "2", "3" and so on, according to their  
corresponding match-group subexpressions in your regex.

> I would think that naming an event parameter would be the
> same as a custom event parameter. Appearently, it's not. I'm really  
> not
> following and I believe it's because I don't have the full knowledge  
> about
> how OpenNMS works.

What I'm saying is that there's currently no way to say "match-group  
number 1 goes in a parameter called 'process'".  Adding that  
functionality to Syslogd would not be theoretically difficult, though.

> Ok. Well, it's useful functionality that might draw people (like  
> myself) to
> OpenNMS.

Yep!

> PS. The automation page has "To access this column data, specify the  
> column
> using the format '${<column name>}'". Looking at the code it looks  
> as if
> this is wrong it should be '%{column name}' (note % instead of $). DS.

I think you may be confusing the syntax used in Notifd's subject and  
text message fields with the syntax used in automations.  If you look  
at a stock copy of vacuumd-configuration.xml, it's pretty clear that $
{_columnName} is what works.

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
In reply to this post by Johan Edstrom
Johan Edstrom wrote
I've extended the use-case for a former contract I worked on to also
if anyone is interested, generate Alarms from 'droves' of Syslog -  
info....

http://www.opennms.org/index.php/Syslogd_Automations
Noticed "'droves' of Syslog - info...". How much drove?
I just made a post about performance issues after enabling the syslogd functionality in OpenNMS.
Currently, the share amount of events basically renders the web ui unusable.

Regards,
Jimisola


Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jimisola Laursen
Administrator
In reply to this post by Jimisola Laursen
Hi again,

The initial problem with all syslog messages having the Node of the syslog-ng and not the originating syslog message server still exists. I tried the syslog-ng configuration on http://www.opennms.org/index.php/Syslogd:

destination opennms {udp("127.0.0.1" port(10514));};
log {source(s_all);destination(opennms);};

but it did not work for me. It complained about s_all not being defined. So, I defined it just to come to the conclusion (after reading up) that syslog-ng does not support the use of an explicit udp(), tcp(), internal() in multiple sources (i.e. internal(), udp() etc can only be used once).

So, I created a destination for OpenNMS (d_opennms) and added it to d_mysql as well as loghost (see below).


Having the right host in the Node field of an event is crucial for me.

Any ideas? Is the problem in syslog-ng.conf or within OpenNMS?
How does OpenNMS set the Node field?

Regards,
Jimisola




#source src { unix-dgram("/etc/log/log"); internal(); }
source src {
        #
        # include internal syslog-ng messages
        # note: the internal() soure is required!
        #
        internal();

        #
        # the following line will be replaced by the
        # socket list generated by SuSEconfig using
        # variables from /etc/sysconfig/syslog:
        #
        unix-dgram("/dev/log");

        #
        # uncomment to process log messages from network:
        #
        #udp();
        #udp(ip("0.0.0.0") port(514));
};

# destination OpenNMS on localhost/seldmon02:10514
destination d_opennms {
    udp("monitor02" port(10514));

    # for debug purpose
    # file("/var/log/syslog-ng.opennms.debug");
};

source net { udp(); };
destination d_mysql {
   pipe("/tmp/mysql.pipe"
   template("INSERT INTO logs (host, facility, priority, level, tag, date,
   time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG',
   '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes));
    };
log {
    source(net);

    destination(d_mysql);
    destination(d_opennms);
};

#Send your own logs to monitor02
destination loghost {
    udp("monitor02" port(514));
};
log {
    source(src);
    destination(loghost);
    destination(d_opennms);
};

Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jeff Gehlbach
In reply to this post by Jimisola Laursen
On Jul 18, 2008, at 6:06 AM, Jimisola Laursen wrote:
> Currently, the share amount of events basically renders the web ui  
> unusable.

That's a real problem with any system that stores events, and the only  
cure for it is to mitigate the flood.  You can do this at syslog-ng  
(highly recommended) by filtering out things that you know are "noise"  
rather than forwarding them to OpenNMS -- think of this as analogous  
to constraining the types of SNMP traps that your Cisco routers send  
to OpenNMS.  Overall, keep in mind that OpenNMS is not meant to be an  
event warehouse.

There's a facility in the OpenNMS trap daemon to let SNMP traps be  
discarded before they get turned into events.  It would be nice if  
Syslogd had a similar facility, but currently I don't think it does.

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: OpenNMS and syslog relaying

Jeff Gehlbach
In reply to this post by Jimisola Laursen
On Jul 18, 2008, at 6:17 AM, Jimisola Laursen wrote:

> The initial problem with all syslog messages having the Node of the
> syslog-ng and not the originating syslog message server still  
> exists. I
> tried the syslog-ng configuration on
> http://www.opennms.org/index.php/Syslogd:
>
> destination opennms {udp("127.0.0.1" port(10514));};
> log {source(s_all);destination(opennms);};
>
> but it did not work for me. It complained about s_all not being  
> defined. So,
> I defined it just to come to the conclusion (after reading up) that
> syslog-ng does not support the use of an explicit udp(), tcp(),  
> internal()
> in multiple sources (i.e. internal(), udp() etc can only be used  
> once).

I'm not sure about this.  I rarely mess with syslog-ng once it's  
working right, and I'm bad about taking notes during the "getting it  
working right" phase.

> Having the right host in the Node field of an event is crucial for me.
>
> Any ideas? Is the problem in syslog-ng.conf or within OpenNMS?
> How does OpenNMS set the Node field?

It's all about the combination of forwarding-regexp, matching-group-
host, and matching-group-message.  Keep in mind that the forwarding-
regexp is evaluated against the syslog message in a fairly raw state  
-- I believe only the PRIO field has been stripped from the beginning  
at the point that the message is checked against this regex.  
Whichever subexpression in your forwarding-regexp matches what should  
be the hostname or IP address of the originating system should have  
its number in the matching-group-host attribute, and whichever  
subexpression matches the body of the message should have its number  
in matching-group-message.

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
12