perfsonar development work

["Jeff W. Boote" <>]

Nicolas Simar wrote:
> Roman Lapacz wrote:
> > Nicolas Simar wrote:
> >
> > > Hi,
> > >
> > > In the RRD MA configuration file, the person who build the metadata 
> > > configuration file  can chose the "Data storage unit".
> > >
> > > The visualisation tools querry the information from the MA.
> > >
> > > What happens if one RRD MA instance export bps, another one kBps?
> > > 1) Do the visualisation tool have to deal with it?
> > > 2) Do the MA have to normalise it?
> > > 3) Are there best practices we should suggest to ensure consistency 
> > > in the service offered globally?
> >
> >
> >
> > I think we should have one terminology for units to avoid the 
> > situation when the same unit will be called differently in two 
> > instances of RRD MA (example bps and bit/s).
> > Transformation between units should be done by transformation service 
> > (TrS). No need to add this functionality to MA (it 's possible but may 
> > increase the time of responses from MA).
>
> So there are possibly two topics:
> - agreeing on unit names
> - what to do in the meanwhile, whilst we don't have any transformation 
> service?

This is really the same problem as the time units in the SQL-MA that we 
discussed last month. The schema example I sent out during the Salvador meeting 
describes one way a request could be made to do this kind of data 
transformation. This adds a fair amount of complexity for questionable benefit.

In my opinion we need two things:

1) Very clear standards-like documents that specifies exactly what the allowable 
values are for every attribute and entity in each schema. (In this case, the 
allowable values for valueUnits in the datum.)

2) We need to decide if we want to allow clients to request things in specific 
units. And if so, how.

There is a trade-off here. In general, there is more compute power available to 
the clients simply due to the distributed nature of many clients per server. 
However, forcing the client to do the data transformation adds lines of code to 
the clients - and distributing complexity is not so nice.

The middle ground is the transformation services (as Roman indicated). However, 
that adds a fair amount of complexity as well. I think the added complexity will 
be worth it for some things, but I'm not sure it is worth it for things like 
units conversion.

At first glance, it would seem that the easiest immediate (perhaps temporary) 
solution is to specify only one or two specific unit types that are allowed in 
the schema. Then have services serve the data as they want (of the allowed types 
of course). Then clients will need to transform if needed. This requires that 
clients support all allowed unit types. (Which is why we should keep the number 
of unit types very small.)

jeff

P.S.

I don't really understand why anyone would be putting anything other than 
octets(bytes) in the RRD files. The SNMP MIB variable is in octets - why mess 
with it? Can someone tell me what other units are being used for utilization, 
and what they are using to collect the data so that it ends up in other units? 
It would help me to understand the scale of this problem - how many possible 
units are in typical use for this metric?