perfsonar development work

["Vedrin Jeliazkov" <>]

Hi Barney,

The issue you're reporting has been discussed indeed in several mail threads
which we believe you've followed. In particular, rrdtool does not compute
averages "on the fly" while responding to queries, but rather stores them in
the predefined RRD file during updates. So, basically it is impossible to
request and get back a resolution, which is not computed and stored in the RRD
file already. In you particular case this means that rrdtool will always
return 60 second resolution, regardless of the selected time interval. A data
set with such resolution covering a long period of time is *huge* and we
believe that it's unreasonable to expect that clients should be able
necessarily to cope with such volume of data. If we take as an example a time
period of one month with a mandatory resolution of 60 seconds, this gives us
2592000 values, to be displayed on a screen with - let's say - 1024 pixels.
There's not much sense in doing this and wasting bandwidth and CPU cycles. If
you'd like to see more details when performing a zoom, the client could of
course (only then) ask for more samples in the shorter time period. But this
doesn't solve the following basic problems:

a) currently there's no way to learn what are the available resolutions for
different periods of time in some measurement archive service;

b) rrdtool cannot compute averages on the fly, so if you have only 60 sec
resolutions for long periods of time, then you would not be able to visualise
anything longer than a few days;

Possible solutions to these problems require some design decisions on the
service side, which would hopefully become available at some point.

Best regards,
Vedrin

Barney Garrett 
<>
 wrote:

> Bringing up an old issue ... I've upgraded to version 0.09 of Perfsonar 
> UI but the problem of out of memory error when selecting timescales 
> longer than 1 week that I was experiencing in version 0.08 is still 
> present.  If you remember I'm somewhat stressing the system ;) since I 
> only have a 60 second resolution in my rrd files.
> 
> Barney
> 
> -------- Original Message --------
> Subject: Re: A bug and some questions regarding perfsonarUI
> Date: Mon, 18 Sep 2006 07:16:05 +0300
> From: Nina Jeliazkova 
> <>
> Reply-To: Nina Jeliazkova 
> <>
> To: 
> 
> CC: 
> 
> References: 
> <>
> 
> Hello,
> 
> Thank you for reporting this. After some tests we can confirm 
> OutOfMemoryError
> is due to 1) graphing unexpectedly large number of data points 2)for larger
> time intervals it is the unabilty of Axis library to handle the response 
> as a
> large XML structure in memory.
> This of course depends on how much memory Java machine is allowed to use 
> (for
> example on my laptop the error occurs when querying for last month, not last
> week).
> 
> Indeed you are right that 60sec step is responsible to the Out of memory
> error. We found that your service always returns data with resolution of 60
> sec, regardless of the resolution in the request. There are two issues here.
> RRD needs the time boundaries to be multiple of the resolution, otherwise it
> returns the data with the finest resolution it has. perfonarUI 0.8 doesn't
> align time boundaries due to a bug and therefore the returned data is  with
> finest resolution. However, the default behaviour of RRD is to keep data for
> recent time intervals with 5 min resolution and for time intervals back in
> time with even larger resolution. This is the reason other available MA
> services return less data points and work fine when queried for week, month
> and year.
> 
> Then we fixed the time boundaries bug in the developer version, but for our
> surprise, your service continued to return data with 60 sec resolution,
> regardless of the provided resolution value. This is definitely a different
> behaviour than other MA services, which return data with the provided
> resolution. This also seems like a discrepancy with the RRDTool 
> documentation.
> Could it be that some particular setting in your RRD MA service makes it to
> ignore resolution?
> 
> The 10 min delay has the following simple explanation. RRD files are by
> default updated on 5 minutes, so when asking for an interval including the
> last 5min,  it happens very often that the service return NaN for data 
> values.
> Therefore we've moved the time interval boundary 5 min back. The next 5 min
> are due to the fact that rrd files for ISTF service are updated on one 
> machine
> (that runs Cacti) and copied every 5 minutes to the other machine that 
> runs MA
> service. To avoid any NaNs in the response, we've moved the time boundary
> another 5 min back (although this might be over pessimistic solution).
> 
> In short, right now there is no way to avoid this 10 min delay, but we could
> provide an option in the next release. Again, there is no way to change
> resolution in user interface, but even if there were, your service is 
> ignoring
> the provided resolution for some reason.
> 
> Thanks again for reporting the bugs. Memory issue definitely needs to be
> addressed and we'll try to find a solution for the next release.
> 
> 
> Best regards,
> Nina