perfsonar development work

["Jeff W. Boote" <>]

Nicolas Simar wrote:
> Vedrin Jeliazkov wrote:
> > Hi Barney,
> >
> > The issue you're reporting has been discussed indeed in several mail 
> > threads
> > which we believe you've followed. In particular, rrdtool does not compute
> > averages "on the fly" while responding to queries, but rather stores 
> > them in
> > the predefined RRD file during updates. So, basically it is impossible to
> > request and get back a resolution, which is not computed and stored in 
> > the RRD
> > file already. In you particular case this means that rrdtool will always
> > return 60 second resolution, regardless of the selected time interval. 
> > A data
> > set with such resolution covering a long period of time is *huge* and we
> > believe that it's unreasonable to expect that clients should be able
> > necessarily to cope with such volume of data. If we take as an example 
> > a time
> > period of one month with a mandatory resolution of 60 seconds, this 
> > gives us
> > 2592000 values, to be displayed on a screen with - let's say - 1024 
> > pixels.
> > There's not much sense in doing this and wasting bandwidth and CPU 
> > cycles. If
> > you'd like to see more details when performing a zoom, the client 
> > could of
> > course (only then) ask for more samples in the shorter time period. 
> > But this
> > doesn't solve the following basic problems:
> >
> > a) currently there's no way to learn what are the available 
> > resolutions for
> > different periods of time in some measurement archive service;
>
> Does the group feel this is a required feature?
> If yes, what advantage/disadvantage would it bring?

This is information that should be in the LS service, or perhaps an LS service 
dedicated to the MA. (Another use for hierarchical-LS!) And, I agree that the 
data resolution information is absolutely critical to expose.

> > b) rrdtool cannot compute averages on the fly, so if you have only 60 sec
> > resolutions for long periods of time, then you would not be able to 
> > visualise
> > anything longer than a few days;
>
> Do you mean there that it should be a feature of the RRD MA, the rrdtool 
> or of a transformation service?

There are 3 ways to address this:
1. Allow the client to ask for other resolutions and make the service transform 
the data into the resolution the client asks for. (Complicated - and in my 
opinion, contrary to the modular architecture.)
2. Provide a transformation service. (Requires resolution information be 
exposed)
3. Transform the data at the client. (Requires resolution information be 
exposed)

I believe the right place for the development to initially happen is at the 
client. But, we need to watch for things like this that are developed at clients 
that we can pull off as modular transformation services. That is where we want 
this functionality in the long-term. (I would prefer not to start developing 
transformation services until we have AA pieces in place - AA integration will 
be the hard part of transformation services, so it would be good to get some 
experience with the more direct cases first.)

jeff