Are there storage options than JRobin's JRD files, .i.e RDBMS?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Are there storage options than JRobin's JRD files, .i.e RDBMS?

Jimisola Laursen
Administrator
Hi!

I haven't been able to find any alternatives to JRobin's JRD files as storage option. Is it possible?
Ideally, I'd like to be able to use a RDBMS such as MySQL or PostgreSQL.

Knowing that there are some plans to redesign and rewrite the web application I think it would be very useful if your users can use e.g. Hibernate when extending (writing plugins) the web application.

I also suspect that performance and flexibility (especially, with queries) would be better with a RDBMS than JRD files.

Regards,
Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Alexander Finger
Jimisola,

the rrd files (or jrd's) store time series data for one year, in
configurable intervals.

>From a general point of view that data is already not "relational",
it's a sequence of values with a time reference.

If you store that in a database the main added value of the database
engine is to put this time series data in a file on the harddrive. And
retrieve it when needed = you use a mysql or psql or whatever to
perform what the RRD storage engine is already doing (and has been
optimized for).

A database engine is really good when you need to get answers to
rather vague questions ("give me the phone number of all people with
an "A" in the firstname which have a white cat and live in a suburbian
area"). That capability is not needed when accessing the performance
(rrd) data for a node - you know already exactly what you want. The
thing to do is to open the file, read (at a certain position) and
close it.

So a rdbms is not adding value in performance (the contrary is the
case, you create overhead) nor in data retrieval or storage.

If you have only few nodes that might not really hurt. If you scale
up, however, the performance of a rdbms stands way behind the
filesystem - mainly because the data structure does not relational at
all. And because it's a lot of data to store on larger systems.

We spent some time thinking this over and did actually a calculation
to proof that using a rdbms would be good to use..but our conclusion
was the contrary ;)

There is however a possibility to retrieve data via a XML interface
from opennms, for integration purposes. The other interesting thought
you bring up is to have JDBC access to the data. An ODBC interface to
the RRD storage engine could be fun to have, but replacing the RRD
logic is hard if not (currently) impossible.

cheers
Alex
On Thu, Sep 4, 2008 at 9:43 AM, Jimisola Laursen <[hidden email]> wrote:

>
> Hi!
>
> I haven't been able to find any alternatives to JRobin's JRD files as
> storage option. Is it possible?
> Ideally, I'd like to be able to use a RDBMS such as MySQL or PostgreSQL.
>
> Knowing that there are some plans to redesign and rewrite the web
> application I think it would be very useful if your users can use e.g.
> Hibernate when extending (writing plugins) the web application.
>
> I also suspect that performance and flexibility (especially, with queries)
> would be better with a RDBMS than JRD files.
>
> Regards,
> Jimisola
> --
> View this message in context: http://n2.nabble.com/Are-there-storage-options-than-JRobin%27s-JRD-files%2C-.i.e-RDBMS--tp838218p838218.html
> Sent from the OpenNMS - discuss mailing list archive at Nabble.com.
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss
>

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Jimisola Laursen
Administrator
Hi

I appreciate your answer.

> There is however a possibility to retrieve data via a XML interface
> from opennms, for integration purposes. The other interesting thought
> you bring up is to have JDBC access to the data. An ODBC interface to
> the RRD storage engine could be fun to have, but replacing the RRD
> logic is hard if not (currently) impossible.

There are cases when one does not want to simply access the data based on time and also make use of more complex SQL queries - which I assume that JRobin does not support. Also, having to handle multiple files instead of one database increases "complexity". Most importantly, using a RDBMS would allow for easier integration with e.g. data mining tools, ORM libraries (= easier data by application/plugin) etc.

I assume that we both agree that having such a tight coupling between business logic and underlying storage layer is less than ideal. Ideally, the user/customer would be able to choose the underlying storage depending on their preferences (performance is not always everything).

Anyway, our team have come to see a lot of OpenNMS benefits (thanks). But the lack of JDBC access is definitely on the major cons as well the configuration "confusion" (e.g. it being split between xml files and web-GUI - sometimes both).

Kind Regards,
Jimisola
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Jeff Gehlbach
On Sep 7, 2008, at 2:40 PM, Jimisola Laursen wrote:

> There are cases when one does not want to simply access the data  
> based on
> time and also make use of more complex SQL queries - which I assume  
> that
> JRobin does not support. Also, having to handle multiple files  
> instead of
> one database increases "complexity". Most importantly, using a RDBMS  
> would
> allow for easier integration with e.g. data mining tools, ORM  
> libraries (=
> easier data by application/plugin) etc.

I know that some work was done during Dev-Jam in late July toward  
making an extremely flexible way to get time-series data.  I was only  
listening on the periphery of some of the conversations and looking at  
some of the whiteboarding, so I don't know the details.  While I'm  
fairly certain it will not have an SQL interface (at least initially),  
this mechanism, when it's completed, should largely abstract away the  
back-end storage.

> I assume that we both agree that having such a tight coupling between
> business logic and underlying storage layer is less than ideal.  
> Ideally, the
> user/customer would be able to choose the underlying storage  
> depending on
> their preferences (performance is not always everything).

We don't like the coupling either.  JRobin actually does have  
pluggable back-ends.  Currently there exist two file back-ends (one  
uses Java's regular file support, the other uses native I/O) and one  
memory back-end.  There's an abstract SQL back-end class as well, but  
I don't know of any working implementations.

> Anyway, our team have come to see a lot of OpenNMS benefits (thanks).

Thanks for recognizing that openNMS doesn't (totally) suck :)

> But the lack of JDBC access is definitely on the major cons

I suspect that the new performance data retrieval framework will  
provide a rich set of DAOs, which depending on the target application  
is even better than JDBC.  I'm sure it could be wrapped in JDBC,  
though, for use with the multitude of report-generation packages out  
there that hook in via JDBC.

> as well the configuration "confusion" (e.g. it being split between  
> xml files and web-GUI
> - sometimes both).

All the configuration is in XML files, it's just that web interfaces  
for editing the configuration have taken a lower priority compared to  
other features that people chose either to work on personally or to  
fund as enhancements.  Someday all configuration will be accessible  
from the web UI; the best way to bring someday closer to today is to  
help out.

-jeff

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Alexander Finger
In reply to this post by Jimisola Laursen
Hi Jimisola,

the problem is the writeback performance. OpenNMS collects heavy
amounts of data which need to be stored away.
We are running datacollection on 2.5k and polling 50k Nodes. A RDBMS
would not be able to cope with the load. If you let users chose you
take high risk of having a system which stalls on writeback.

If I was to set up a business-warehouse-like system to do strategic
analysis like you point out my choice would probably be to extract the
key performance indicators regularily via xml (and odbc for outages)
and feed the cubes from that.

We actually do that, I think.

cheers
Alex

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Les Mikesell
In reply to this post by Jimisola Laursen
Jimisola Laursen wrote:

>
>> There is however a possibility to retrieve data via a XML interface
>> from opennms, for integration purposes. The other interesting thought
>> you bring up is to have JDBC access to the data. An ODBC interface to
>> the RRD storage engine could be fun to have, but replacing the RRD
>> logic is hard if not (currently) impossible.
>
> There are cases when one does not want to simply access the data based on
> time and also make use of more complex SQL queries - which I assume that
> JRobin does not support. Also, having to handle multiple files instead of
> one database increases "complexity". Most importantly, using a RDBMS would
> allow for easier integration with e.g. data mining tools, ORM libraries (=
> easier data by application/plugin) etc.

Data mining and analysis tools often recommend extracting your live data
into a different database even when the live copy is already in sql,
partly to give you a chance to clean up the data and denormalize the
tables and partly to avoid performance impact on the live database.

> I assume that we both agree that having such a tight coupling between
> business logic and underlying storage layer is less than ideal. Ideally, the
> user/customer would be able to choose the underlying storage depending on
> their preferences (performance is not always everything).

Or, have a way to snarf the data you want periodically and store it the
way you want.  For example, I'd like to track the max traffic on a few
dozen internet-facing interfaces, but once a day would be good enough so
it is probably feasible with the URL based queries.  I started to test
that but I'm having some other performance problems and don't have a
working example yet.

> Anyway, our team have come to see a lot of OpenNMS benefits (thanks). But
> the lack of JDBC access is definitely on the major cons as well the
> configuration "confusion" (e.g. it being split between xml files and web-GUI
> - sometimes both).

A nifty solution to this might be to glue jrobin into the kettle (now
Pentaho Data Integratin but still GPL) tool that can copy and convert
among a wide variety of other formats/databases.  http://kettle.pentaho.org/

It might already be possible to use its xml reader but it is probably
easier in perl with something like XML::Twig and DBI.  However, having
direct access to the data in jrobin might make it possible to do some
more intelligent things like smoothing momentary spikes, calculating
95th percentile values, and correctly computing the max bandwidth used
over a long interval where failover between multiple interfaces has
happened.

--
   Les Mikesell
    [hidden email]





-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Les Mikesell
In reply to this post by Alexander Finger
Alexander Finger wrote:
>
> If I was to set up a business-warehouse-like system to do strategic
> analysis like you point out my choice would probably be to extract the
> key performance indicators regularily via xml (and odbc for outages)
> and feed the cubes from that.
>
> We actually do that, I think.

Do you have any code examples or hints about tools that work well for
this that you can share?

--
   Les Mikesell
    [hidden email]


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

David Hustace
In reply to this post by Jimisola Laursen

On Sep 7, 2008, at 2:40 PM, Jimisola Laursen wrote:

Anyway, our team have come to see a lot of OpenNMS benefits (thanks). But
the lack of JDBC access is definitely on the major cons

A RESTFul API is on its way.


as well the
configuration "confusion" (e.g. it being split between xml files and web-GUI
- sometimes both).

Well, the GUI is just the front-end to the configuration be it stored in a file or the DB.  Most integrators want access to configuration and not simply a UI and openNMS strives to maintain this flexibility (with greater flexibility comes greater complexity).

I do agree that there is a lack of consistency in this area but it is mainly within the configurations and UIs themselves, of which, I'm as much to blame as anyone.  An architectural goal is to get his cleaned up for a 1.6.1+ release... which means, after we get a 1.6 release ASAP.

<bias>I will argue, however, that once someone has invested the time to gain a comprehensive understanding of the openNMS platform and get beyond the initial complexity, the benefits are worth that initial investment.  There are things that can be done simply through configuration alone that one would find difficult if not impossible to do with other management systems.</bias>


David Hustace
The OpenNMS Group, Inc.


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Alexander Finger
In reply to this post by Les Mikesell
I will have a look and see what is share-able.
cheers
Alex

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Are there storage options than JRobin's JRD files, .i.e RDBMS?

Jimisola Laursen
Administrator
Hi all,

Sorry about stiring it up like this :)

To clear things out, our main usage is to access the storage (here: JRobin) from a web application - not from a strategic analysis/data mining tool. But, using JRobin does not allow us to use easily use Hibernate nor plain old SQL. Also, we need the current data and not a "minute-old" data. Hence, exporting and importing via XML is not solution for us. OpenNMS is apparently great at collecting data, but it would be even greater if the data was easier to access and integrate with existing web application (framework) solutions.

Kind regards,
Jimisola