perfsonar development work

["Jeff W. Boote" <>]

Nicolas Simar wrote:
> Hi,
>
> I renamed this thread as it is not about schema version. I summarise the 
> information seen on the mailing list.
>
> 1. What is the problem
>
> The schema (protocol?) evolves over time. The deployment of services by 
> the networks operators don't always follow this evolution. The 
> visualisation tools are ending-up having to deal with hard-coding a tool 
> deployed and the schema used.
> (Temporary measure exist to give some information through the URL, but 
> not for all the services).

It is not a problem that the schema evolves over time. It was designed to evolve 
over time. This is a strength, not a problem. Otherwise, we could have just as 
easily have used binary protocols for client/service interactions. We picked XML 
specifically because we wanted something extensible.

The problem is that there are multiple levels of information that need to be 
communicated, and they are not being adequately communicated. Only one of those 
things is 'schema' (or schema version). The others are protocol and profile. 
(See attached message.)

I'm not convinced that a message-parameter is really enough here for the 
'general' solution. I suspect we may want this information published to the LS 
somehow as well. But, I guess something like this would at least be a start.

I'm also not sure a message parameter is the best way to expose this from the 
service... It was just the least intrusive way I could think of. Perhaps someone 
else has a better idea?

jeff

> 2. Solutions
>
> 2.1. Protocol extension (as already proposed by Jeff) to support picking 
> up of protocol version(s) supported.
>
> 2.2. Have the web-services URL containing the product or protocol 
> version number.
>
>
> 3. Benefits
>
> 3.1 Protocol Extension
> This would allow the visualisation client to deal with protocol 
> versioning automatically.
> - PerfsonarUI will be able to abandon MA.conf (combined with the 
> deployment of the distributed LS).
> - the web-services could adapt their answer with the client or the other 
> web-service version (useful as the protocol evolves and the products 
> might not deployed as frequently).
>
> This is an assumption: do you validate it???
>
> - Other benefits examples???
>
>
> 3.2. URL containing the (protocol/schema) version number.
> It is a temporary hack to provide the information (actually, the hack is 
> to provide the web-service version number).
> Ideally, the URL should not change over time for a deployed instance (so 
> that static configuration files don't have to be change by the users 
> each time there is a service upgrade).
> However
> (a) the web-service name might change over time (for what reasons?),
> (b) it is expected to use the LS to be communicated the URL when some 
> data are needed (thus the URL could change over time).
> But we should maybe not expect all the visualisation clients to rely on 
> the LS (no case known to me so far though) and try not to have too 
> "dynamic" service URLs (thus removing the product version number isn't a 
> solution).
>
> 3.3. One could expect that within a year or two, all the deployed 
> instances would have moved towards a newer web-service version (with 
> protocol information included). Combined with the distributed LS, it 
> would imply that the visualisation client wouldn't need static files 
> anymore (which doesn't mean that all of them would abandon the static 
> files).
>
> 5. Question
>
> - Do we want to have a versioning for the schema or for the protocol 
> (yet currently undefined, but in the process to)?
>
> - Where shall this information be located?
> In any messages exchanged? Only in the messages sent by the LS? In a new 
> type of message sent by any web-service when requested?
>
> - What impact will it have on the web-services?
>
> - Jeff, can you re-sent your proposal for adding versioning? I don't 
> remember in which thread it was sent.
>
>
> Cheers,
> Nicolas
>
> Roman Łapacz wrote:
> > Nina Jeliazkova napisał(a):
> >
> > > Hi all,
> > >
> > >   
> >
> > Hi
> >
> > > I fully support Loukik message, that ideally, the protocol should be
> > > responsible to reveal versioning information.   
> >
> > In the comming release of rrd ma you will find new type of request 
> > LookupInfoRequest/Response which help you in getting general service 
> > information (including service version number; we could add protocol 
> > version to this set as well). The implementation of this message type 
> > is located in the base so each service can have it.
> >
> > Take a look at the documantation of it:
> > https://svn.perfsonar.net/svn/perfsonar/tags/GEANT2-JAVA-RRD-MA-2.3-RC4/doc/Interface_Specification.doc 
> >
> >
> > Of course I also support the idea of having protocol version inside 
> > requests which could instruct services how to answer (what protocol 
> > implemetation to use)
> >
> > Roman
> >
> >
> > > Regarding perfsonarUI, the lack of such information so far led to 
> > > introduction
> > > of "supported schema" field in MA.conf. One of the factors delaying 
> > > abandoning
> > > MA.conf in favor of lookup service, is that after such transition, 
> > > the version
> > > information will be lost.
> > > I see versioning in URL as a temporary workaround, to become 
> > > irrelevant when
> > > the protocol is updated to support versioning.
> > >
> > > Best regards,
> > > Nina
> > >
> > >
> > > Loukik Kudarimoti 
> > > <>
> > >  wrote:
> > >
> > >  
> > >
> > > > Nicolas,
> > > >
> > > > The lack of protocol versioning and a method to discover supported 
> > > > protocols required us to make this choice.
> > > >
> > > > Without such versioning in the URL, its difficult to know which 
> > > > version of software has been installed. If the protocol can be 
> > > > extended (as already proposed by Jeff) to support picking up of 
> > > > protocol version(s) supported, this can be safely removed. Ideally, 
> > > > this protocol should also help in picking up the version of software 
> > > > deployed.
> > > >
> > > > regards,
> > > > Loukik.
> > > >
> > > > Nicolas Simar wrote:
> > > >   
> > > > > Hi Loukik, Michalis
> > > > >
> > > > > Can you tell about the rationale for having the release version name?
> > > > >
> > > > > Hi Nina and Andreas,
> > > > >
> > > > > Would you prefer to limit the URL changes of the web-services or do 
> > > > > you prefer to have the product version in the URL?
> > > > >
> > > > > Cheers,
> > > > > Nicolas
> > > > >
> > > > > Maciej Glowiak wrote:
> > > > >     
> > > > > > Nicolas Simar wrote:
> > > > > >
> > > > > >       
> > > > > > > Hi,
> > > > > > >
> > > > > > > looking at the web-services names
> > > > > > >
> > > > > > >           
> > >
> > > http://rrd-ma.perfsonar.xxx:8080/perfSONAR-RRD-MA-2.2/services/MeasurementArchiveService 
> > >
> > >  
> > >
> > > > > > > I see there is a part mentioning the service version(2.2). Is 
> > > > > > > there a
> > > > > > > particular benefit of seeing the service version?
> > > > > > >
> > > > > > > I am wondering about this, because when the service will be 
> > > > > > > updated, the
> > > > > > > version will be updated and the tools won't be able to access it 
> > > > > > > until
> > > > > > > they are updated.
> > > > > > > It this statement correct?
> > > > > > >           
> > > > > >
> > > > > >
> > > > > > Nicolas,
> > > > > >
> > > > > > AFAIR it was a requirement from Release Mgmt. Team for perfSONAR 2.0.
> > > > > > You should contact Loukik or Luis in order to get more information 
> > > > > > about
> > > > > > the reason for using version number.
> > > > > >
> > > > > > Anyway, Ilias and Roman are right - that's default name (contaning
> > > > > > version number) and everybody can change it after the installation.
> > > > > >
> > > > > >         
> > > > >
> > > > >
> > > > >
> > > > >       
> > >
> > >
> > >
> > >
> > >

--- Begin Message ---

• From: "Jeff W. Boote" <>
• To: Loukik Kudarimoti <>, "" <>
• Cc: Roman Lapacz <>, Michael Michalis <>, Szymon Trocha <>, , Nicolas Simar <>, Jason Zurawski <>
• Subject: Re: [PS-relmgmt] Re: [Bug 221] RRD MA 2.3 backwards compatibility problems with 2.2
• Date: Tue, 25 Sep 2007 11:12:31 -0600

Hi All,

I'm bouncing this relmgmt message out to the full development list, because I think it is something we need to address more firmly across the full development effort. Not just within the release management team.

I want to propose a direction for future developments. (Note, I'm not proposing the solution for this particular release. I'll leave that up to the relmgmt team. I'm just seeing the same problem come up in different forums, and this seems like a good example to illustrate a possible solution.)

I agree with the directions Roman has gone here. These changes are needed to maintain consistency between multiple data types as Roman has suggested. However, it is clear that the changes has caused some interoperability problems with clients. So, we need a better migration strategy for these kinds of modifications in the future.

Jason and I talked about this particular problem some because it also effects the pS-PS SNMP MA. (And we have had some discussion about other similar problems in our pS-PS development team.)

In a discussion we had yesterday, Martin suggested that there are three levels of things that need to be standardized in some way:

1) schema
2) protocol (generic service level)
3) profile (specific data types in a service)

I agree with this. But, we do not adequately have this represented in our client/service interactions (or documentation).

Currently, the 'schema' level is described in our messages (using the
eventType).

The 'protocol' level is not really described yet, but it seems clear that it needs to be.

I *think* the 'profile' can be adequately described by a combination of 'schema' and 'protocol'. (But, we can add something more later if this turns out to be false.)

I would like to propose the following for the future:

A message level parameter to indicate the 'protocol' type and level.

<nmwg:parameters>
<nmwg:parameter name="pSprotocol">
http://perfsonar.net/ns/protocol/MA/${date}
</nmwg:parameter>
</nmwg:parameters>

Clients would specify the version they expect the service to use. Services should report back the version they are using or a response code indicating the requested protocol level is unsupported (preferably with a supported range of protocol versions). [Response codes must be given in a way that is acceptable to the client. This is one area where backwards compatibility will need to run a long ways.]

In this way, a service can support a specific range of protocol message semantics. And it has a way to indicate how much backwards compatibility it will support.

This could be used to deal with this current backwards compatibility issue in this way:

If the 'client' sets this message level parameter, then the RRD MA could respond with the datum in snmp:datum elements, if not - then it can use nmwg:datum. (Likewise for the other issues described here.)

jeff

Loukik Kudarimoti wrote:

Roman,

As highlighted in bugs 221, 222, there are backwards incompatible changes to the way in which utilization data is retrieved from the new version of RRD MA. I suggest a phone conference tomorrow to plan on resolving these issues

Suggested time: 12:00 hrs UK time, 13:00 hrs PSNC time
Expected attendees: Loukik, Luis(?), Michalis, Szymon, Roman, Nicolas

thanks,
Loukik.

Roman Lapacz wrote:

Loukik Kudarimoti wrote:

Hi Roman, Szymon,

Hi

I welcome the discussions on schema and changes being made in the trunk of RRD MA. What I don't understand is why it has been decided to have these schema changes released as part of the update at this stage. It clearly breaks backwards compatibility.

I am aware that Nicolas wanted new metrics to be released as part of perfsonar-bundle 2.2 version but there was no mention/need to release these schema changes. Right?

For those new metrics the namespaces of datum elements are tied to eventTypes so the utilization should follow the same approach to keep consistency between messages (messages for utilization, errors and discards are very similar).

Replacing supportedEventType parameter is a change which effects metadata service configuration so it should not harm the communication with clients.

Roman

Loukik.



wrote:

http://bugzilla.perfsonar.net/show_bug.cgi?id=221





------- Comment #1 from

2007-09-24 08:40 -------
(In reply to comment #0)

Serious ones:
1. As an effect of removal of parameter SupportedEventTypes in the metadata
config, some of the service responses have been altered. Following requests in
the interface spec for RRD MA 2.2 have been modified and thus resulted in
incompatibilities:
- Page 13, example 2 (compare with Page 16, example 2 in 2.3 spec)
- Page 18, example 8 (compare with Page 21, example 8 in 2.3 spec)
- As a result LS registration information format changes as well.

supportedEventType parameter in metadata configuration file is not supported
any more. eventType element has replaced it. This was agreed with Jason and
Jeff on dev mailing list some time ago and it's used in perl implementation as
well.

2. Chained requests not supported any more
- Pages 21- 27, examples 11, 12, 13, 14 are not supported anymore

3. Even though the documentation states that the response datum for utilisation
uses nmwg namespace and the datum for drops and discards uses an event type
based namespace, the implementation is different. All datum returned use a
namespace defined by the eventType. This raises serious backwards
incompatibility with 2.2

Documentation has been updated. Namespace of datum element is tied to eventType
element (agreed with Jason and Jeff).

Less serious ones:

4. Key contents have been changed (But keys are expected to be opaque. So, it
shouldn't matter. We need to make it known to other dev teams though)

select parameter elements are not put in parameters element of nmwg namespace.
They are grouped according to namespace.

5. For utilisation, both namespaced and non-namespaced eventTypes were
supported in 2.2. This *seems* to have been changed to support only namespaced
eventTypes. This, we think, is an acceptable deprecation and *should* have
minimum impact on the client developers

Right. We want to support only namespaced eventTypes and this was announced
some time ago on mailing list by Szymon.

--- End Message ---