perfsonar development work

["Jeff W. Boote" <>]

Roman Lapacz wrote:
> Jeff W. Boote wrote:
> > Roman Lapacz wrote:
> > > HI Nina and Vedrin,
> > >
> > > Vary good points!
> > >
> > > What do you think to take this simple solution: just putting 
> > > parameters of rrdcreate in the keys in the rrd metadata configuration 
> > > file.
> > >
> > > example:
> > >
> > > <nmwg:data id="data1" metadataIdRef="meta1">
> > > <nmwg:key>
> > > <nmwg:parameters>
> > > <nmwg:parameter 
> > > name="file">/opt/perfsonar-20060724/data/rrd/test/test.rrd</nmwg:parameter> 
> > >
> > > <nmwg:parameter name="dataSource">bytes</nmwg:parameter>
> > > <nmwg:parameter name="valueUnits">Bps</nmwg:parameter>
> > > <nmwg:parameter name="step">60</nmwg:parameter>
> > > <nmwg:parameter name="rra">RRA:AVERAGE:0.5:1:1147680 
> > > RRA:MAX:0.5:1:1147680 RRA:MIN:0.5:1:1147680</nmwg:parameter>
> > > <nmwg:parameter name="dataSourceType">COUNTER</nmwg:parameter>
> > > </nmwg:parameters>
> > > </nmwg:key>
> > > </nmwg:data>
> > >
> > >
> > > MetadataKeyResponse message would contain all this information and it 
> > > would be up to the client app how to use it.
> > >
> > > What do you think?
> >
> > And what happens if I store my utilization data in an SQL archive.
>
> The same way. Parameters in the key describing stored data.
>
> Using such parameters is my first thought to solve the problem but you 
> are right that we need to think on some standard solution.

Parameters is fine. Key's are not. Key's are intended to be opaque from the 
client perspective. Any solution that requires a client to understand the format 
of a key is contrary to the current design of the protocol. (And would make 
interoperability extremely difficult.)

> > This would force every client to understand my internal SQL table 
> > representation to determine the resolution and periodic nature of my 
> > data. (More specifically, it would force the clients to have to 
> > understand a different way to get this information from every single 
> > service that exports utilization data from a different type of 
> > archive.) The point of the schema should be to come up with a standard 
> > way to represent this information. Then the rrd fields could be used 
> > to populate the 'standard' ones.
> >
> > This sounds like a really good first example of a protocol that should 
> > be standardized as we have discussed this week.
> >
> > Along these lines, there used to be examples of chaining that 
> > represented the statistics that were available using a filter-chained 
> > metadata. Unfortunately I have not been able to find them, and I don't 
> > want to concentrate too hard on it now during the meeting. I think 
> > they were removed because the current services could not understand 
> > them and they were not kept up to date with the namespace changes. 
> > But, I still think that was the more appropriate way to represent this 
> > information.
> >
> > Anyone have copies of the example XML files that used statistics in 
> > filter chaining somewhere? If I remember correctly we were working on 
> > it in Poznan last year... I will keep looking - but I will probably 
> > have to dig into the old CVS to find it.
>
>
> If I remember correctly we were working in Poznan mainly on 
> SetupDataRequest-FilterRRDSelect-2.xml and 
> SetupDataResponse-FilterRRDSelect-2.xml. I don't think we removed any 
> message examples from CVS.

Yes, this is the example I was looking for. In the 'select' metadata there is 
'resolution'. It was my understanding that this was intended to map to 'step' in 
an rrd. I believe the problem is really on the side that the client can not 
easily find out the 'available' resolutions. If the client requests a resolution 
that is not available, it could return an error response.*

It is my opinion that services should not be computing derived resolution 
datasets on the fly - they should serve what they have. If we want a service to 
compute derived datasets - the 'transformation' should be described (and 
requested). Note: I'm not saying that it could not be implemented as part of the 
same service, just that this type of transformation should be represented by the 
metadata chaining. So, it is clear when derived data is being used - and when 
more direct data is being used.

We should solve this problem not by making the data request/response more 
feature-full, but by giving the client enough information to make a fully 
specified request. (Say by creating another request/response message similar to 
the LSquery/response - but for MA's. And have it use normal metadata/parameter 
sets to describe the data.) Then clients can find out what a service holds.

I understand the key request has been used for this, and I understand why - 
since we have not had reasonable information services. But, if a client is 
looking inside a key - interoperability is being lost. (Clients should not need 
to understand the internal storage representation of data in an archive.) The 
key was intended to be an optimization for repeated queries - not an information 
service like it is currently being used for.

jeff