perfsonar development work

[Nicolas Simar <>]

Vedrin Jeliazkov wrote:
> Hi Barney,
>
> The issue you're reporting has been discussed indeed in several mail threads
> which we believe you've followed. In particular, rrdtool does not compute
> averages "on the fly" while responding to queries, but rather stores them in
> the predefined RRD file during updates. So, basically it is impossible to
> request and get back a resolution, which is not computed and stored in the RRD
> file already. In you particular case this means that rrdtool will always
> return 60 second resolution, regardless of the selected time interval. A data
> set with such resolution covering a long period of time is *huge* and we
> believe that it's unreasonable to expect that clients should be able
> necessarily to cope with such volume of data. If we take as an example a time
> period of one month with a mandatory resolution of 60 seconds, this gives us
> 2592000 values, to be displayed on a screen with - let's say - 1024 pixels.
> There's not much sense in doing this and wasting bandwidth and CPU cycles. If
> you'd like to see more details when performing a zoom, the client could of
> course (only then) ask for more samples in the shorter time period. But this
> doesn't solve the following basic problems:
>
> a) currently there's no way to learn what are the available resolutions for
> different periods of time in some measurement archive service;

Does the group feel this is a required feature?
If yes, what advantage/disadvantage would it bring?

> b) rrdtool cannot compute averages on the fly, so if you have only 60 sec
> resolutions for long periods of time, then you would not be able to visualise
> anything longer than a few days;

Do you mean there that it should be a feature of the RRD MA, the rrdtool 
or of a transformation service?

Cheers,
Nicolas

> Possible solutions to these problems require some design decisions on the
> service side, which would hopefully become available at some point.
>
> Best regards,
> Vedrin
>
> Barney Garrett 
> <>
>  wrote:
>
>
> > Bringing up an old issue ... I've upgraded to version 0.09 of Perfsonar 
> > UI but the problem of out of memory error when selecting timescales 
> > longer than 1 week that I was experiencing in version 0.08 is still 
> > present.  If you remember I'm somewhat stressing the system ;) since I 
> > only have a 60 second resolution in my rrd files.
> >
> > Barney
> >
> > -------- Original Message --------
> > Subject: Re: A bug and some questions regarding perfsonarUI
> > Date: Mon, 18 Sep 2006 07:16:05 +0300
> > From: Nina Jeliazkova 
> > <>
> > Reply-To: Nina Jeliazkova 
> > <>
> > To: 
> >
> > CC: 
> >
> > References: 
> > <>
> >
> > Hello,
> >
> > Thank you for reporting this. After some tests we can confirm 
> > OutOfMemoryError
> > is due to 1) graphing unexpectedly large number of data points 2)for larger
> > time intervals it is the unabilty of Axis library to handle the response 
> > as a
> > large XML structure in memory.
> > This of course depends on how much memory Java machine is allowed to use 
> > (for
> > example on my laptop the error occurs when querying for last month, not last
> > week).
> >
> > Indeed you are right that 60sec step is responsible to the Out of memory
> > error. We found that your service always returns data with resolution of 60
> > sec, regardless of the resolution in the request. There are two issues here.
> > RRD needs the time boundaries to be multiple of the resolution, otherwise it
> > returns the data with the finest resolution it has. perfonarUI 0.8 doesn't
> > align time boundaries due to a bug and therefore the returned data is  with
> > finest resolution. However, the default behaviour of RRD is to keep data for
> > recent time intervals with 5 min resolution and for time intervals back in
> > time with even larger resolution. This is the reason other available MA
> > services return less data points and work fine when queried for week, month
> > and year.
> >
> > Then we fixed the time boundaries bug in the developer version, but for our
> > surprise, your service continued to return data with 60 sec resolution,
> > regardless of the provided resolution value. This is definitely a different
> > behaviour than other MA services, which return data with the provided
> > resolution. This also seems like a discrepancy with the RRDTool 
> > documentation.
> > Could it be that some particular setting in your RRD MA service makes it to
> > ignore resolution?
> >
> > The 10 min delay has the following simple explanation. RRD files are by
> > default updated on 5 minutes, so when asking for an interval including the
> > last 5min,  it happens very often that the service return NaN for data 
> > values.
> > Therefore we've moved the time interval boundary 5 min back. The next 5 min
> > are due to the fact that rrd files for ISTF service are updated on one 
> > machine
> > (that runs Cacti) and copied every 5 minutes to the other machine that 
> > runs MA
> > service. To avoid any NaNs in the response, we've moved the time boundary
> > another 5 min back (although this might be over pessimistic solution).
> >
> > In short, right now there is no way to avoid this 10 min delay, but we could
> > provide an option in the next release. Again, there is no way to change
> > resolution in user interface, but even if there were, your service is 
> > ignoring
> > the provided resolution for some reason.
> >
> > Thanks again for reporting the bugs. Memory issue definitely needs to be
> > addressed and we'll try to find a solution for the next release.
> >
> >
> > Best regards,
> > Nina

--
Nicolas
______________________________________________________________________

Nicolas Simar
Network Engineer

DANTE - www.dante.net

Tel - BE: +32 (0) 4 366 93 49
Tel - UK: +44 (0)1223 371 300
Mobile: +44 (0) 7740 176 883

City House, 126-130 Hills Road
Cambridge CB2 1PQ
UK
_____________________________________________________________________