perfsonar development work

[Guilherme Fernandes <>]

Guilherme Fernandes wrote:
> Jason Zurawski wrote:
> > Verena;
> Hi Jason,
> > > > > > > We are discussing what information services should register into 
> > > > > > > LS,
> > > > > > > besides service URL.  Recently, Jason advised to recognize service
> > > > > > > type by eventtypes, rather than by arbitrary strings like one found
> > > > > > > in "servicetype" parameter.
> > > > > > >
> > > > > > > This seems to work fine with RRD MA services, but not all other
> > > > > > > services submit information about eventtypes when registering to 
> > > > > > > LS.
> > > > > > > Is there a guidance what schema should be supported in this case?
> > > > > > >           
> > > > > > Could you be more specific? Which services are not registering
> > > > > > eventTypes? This disrupts the dLS (and gLS) summarization model as
> > > > > > well...
> > > > > >         
> > > > > So far I am aware of flowsa MA, developed at Surfnet ,  and  BWCTL 
> > > > > and
> > > > > Hades by Win-Labor.  Verena is going to add support for eventtypes,
> > > > > but was asking of the syntax to be supported. Could you please advise
> > > > > her?
> > > > >       
> > > > I am not sure I understand the request, services should be registering
> > > > with the LS in a manner similar to the RRD MA etc. where they 
> > > > register a
> > > > Service metadata description, and the metadata elements they are
> > > > personally responsible for.  These metadata elements *should* include
> > > > eventType elements for the particular data they pertain to.  The
> > > > examples that can be examined are related to the RRD MA (this is the
> > > > only one I personally know of, perhaps there are more).
> > > >
> > > >     
> > > Good to know. In the past, I undertook several attempts, to 
> > > implement something based on an example of another service. However, 
> > > this did not work in a lot of cases, as examples tend to be outdated 
> > > sooner or later.
> > >
> > > It would be a good idea, if anyone expects services to implement 
> > > based on an example, to state this clearly which example, and if 
> > > this is going to be the one which is specified (hopefully) sooner or 
> > > later.
> > >   
> >
> >
> > I am attaching two examples from perfSONAR-PS services (SNMP MA and 
> > perfSONAR-BUOY).  These metadata elements are taken directly from the 
> > store.xml files for each service respectively (making LS registration 
> > trivial).  If your service is not using eventTypes currently, I would 
> > suggest placing them in your storage, or injecting them into the 
> > LSRegistration message.
> >
> >
> > > > > > I think we should concentrate effort to fix this on the service
> > > > > > registration side, not by adding heuristics on the client side 
> > > > > > (which
> > > > > > would then need to be copied to every other client).
> > > > > >         
> > > > > Yes, sure - we've already introduced some perfsonarUI code that 
> > > > > relies
> > > > > on eventtypes registered in LS and are trying to figure out how it
> > > > > will work with services other than RRD MA.
> > > > >       
> > > > I can't really stress this enough: discovery will not work unless
> > > > services are registering in the same manner.      
> > >    ^^^^^^^^^^^^^^^^^^^^^^^^^^
> > >
> > > But that's exactly the point! All we want to know, is HOW this 
> > > should look like.  We cannot implement it in the same manner, if we 
> > > do not know this.
> > >   
> >
> >
> > Perhaps I am missing something, but isn't there documents provided in 
> > the XML LS that describe what is expected?  The general idea is to 
> > include a service description, and the metadata they manage.  It 
> > would be impossible to enumerate what every single service must 
> > provide, but the basic idea is to include measurement metadata 
> > containing at a minimum a subject with a topological element as well 
> > as eventType(s).  This is what the various perfSONAR-PS services 
> > provide, and the RRD-MA does as well.
> >
> > > > If a service does not make available to the LS:
> > > >
> > > > a) Address information (ipv4 or perhaps ipv6)
> > > > d) Domain information (fully qual hostname will do)
> > > > c) any and all possible eventTypes
> > > >
> > > > Discovery will be meaningless.  If service authors have questions or
> > > > concerns about this, lets start talking about it very soon, and if you
> > > > haven't done so *please* read the LS API document, this should explain
> > > > everything:
> > > >
> > > > http://anonsvn.internet2.edu/svn/nmwg/trunk/nmwg/doc/dLS/gLS/phase_1_color. 
> > > >
> > > > html
> > > >
> > > >     
> > > Good to know this one also. It is not obvious to me (and considering 
> > > that quite a lot of other services have not implemented this, it 
> > > seems to be not obvious to others, too) that the API for dLS is 
> > > relevant for implementing the API to LS! Sorry, but how do you 
> > > expect me and others to find exactly this document if noone points 
> > > out that it is the one you need to implement when properly 
> > > registering to LS?
> > >
> > > By the way, I had a look into it. Unfortunately it does not point to 
> > > a schema definition for the elements to be registered with a normal 
> > > LS. So, still, I do not know, how I should do this exactly.
> > >
> > > It's not that I need a discussion what elements should be 
> > > registered. I just need to know which elements you need, and in what 
> > > format I should register them, that's all.   
> >
> >
> > I think you are confusing two issues here: what is necessary for 
> > services to register to LS instances and what LS instances need to do 
> > to distribute this information via the gLS mechanism.  The gLS 
> > document I pasted above does not deal with the former and I do 
> > believe descriptions exist for this task exist already and Maciej 
> > should be able to point people to the exact locations.
> >
> > The requirements for services registering to the LS have not changed 
> > since the original LS was released some time ago.  We have always 
> > asked that each service provide a service information description as 
> > well as metadata that they manage.  In the provided metadata it is 
> > assumed that eventType(s) as well as the necessary domain and address 
> > information is provided (each time I helped design new metadata 
> > descriptions for the various services we ensured that this 
> > information was available).
> This is very clear in the case of MAs, because they handle specific 
> metadata. I don't see it as clear in the case of MPs which, for 
> instance, permit on-demand measurements to anywhere (and, in some 
> cases, from anywhere). What I think that Verena is trying to say is 
> that we don't have a specific metadata definition of what the service 
> does, like you have in your store.xml.
>
> We have eventTypes which are accepted by the service and the schema 
> that defines what is permitted and needed in the requests that use 
> them. We wouldn't register something as complete as you have in the 
> request for perfSONAR-BUOY, because we don't have specific  
> information, i.e. parameters or endpoints. Using your example, we 
> could adapt it to be generic, which would actually be registering the 
> RNC for the requests accepted, and I know this is not what is 
> expected, or we could make it as simples as, i.e. for CL-MP:
>
> <nmwg:data id="data-1" metadataIdRef="serviceLookupInfo">
>    <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"; 
> id="metadata-1">
>      
> <nmwg:eventType>http://ggf.org/ns/nmwg/tools/ping/2.0</nmwg:eventType>
>      
> <nmwg:eventType>http://ggf.org/ns/nmwg/tools/traceroute/2.0</nmwg:eventType> 
>
>      
> <nmwg:eventType>http://ggf.org/ns/nmwg/tools/bwctl/2.0</nmwg:eventType>
>      
> <nmwg:eventType>http://ggf.org/ns/nmwg/tools/owamp/2.0</nmwg:eventType>
>      
> <nmwg:eventType>http://ggf.org/ns/nmwg/tools/cron/2.0</nmwg:eventType>
>    </nmwg:metadata>
>  </nmwg:data>
>
> This would be just registering all eventTypes accepted by the service, 
> and I believe it might satisfy the requirements of the dLS. But we get 
> to another point, as I was mentioned before, it might also be 
> interesting to have not only the functionalities (eventTypes) 
> available, but also extra information about them (which actually means 
> metadata information, and could just be defined using parameters).
Sorry, just realized I was talking specifically about the CL-MP. CL-MP 
uses eventTypes also as functionalities, but this isn't the case for any 
of the other services I know. The other services use the message types. 
In which case we can separate what I said in two parts, the one above 
this, which related to registering which eventTypes the service actually 
deals with and can be done as proposed, and the one below which is more 
about registering that kind of information to the LS or not.

Maybe the information about functionalities hasn't been deemed 
necessary, I know that for AA the client is supposed to make the request 
and then see if it needs authentication or not. For CL-MP, there are 
other things to be considered, i.e. scheduling restrictions, but they 
might also just be done in a per request basis instead of registering it 
to the LS. So if this is the case, we can forget about the paragraph below.

Guilherme
> But these are not related directly to the eventType as action, but to 
> the eventType as functionality. In the sense that you have parameters 
> for defining a ping measurement (i.e. count, size), which is defined 
> by the ping namespace. Now we would also need parameters for defining 
> information about actually using the funcionatility (doing a ping), 
> i.e. if it needs authentication or not. We could define a new 
> namespace for this kind of information, and I believe this is what 
> Verena was actually asking for (not exactly for this purpose).
>
> I don't think there has been any definition of this before, I think it 
> has always been just "anyText". But maybe I'm wrong, it would be good 
> if Maciej jumped into the discussion.
>
> Guilherme
> > The gLS requirements (and the API specification in the above 
> > document) rely on the same sane registration practices, and should 
> > *not* require additional service modifications to function properly 
> > (we designed it this way on purpose to minimize the impact on service 
> > designers in the middle of release cycles).  If a service is not 
> > registering properly, this will damage discovery and the goal of 
> > distributing information.
> >
> >
> > > To summarize: I have absolutly no concerns with the content of a 
> > > registration message.  All I want to know is the proper definition 
> > > of this.
> > >
> > > I'm not sure, where the RRDMA got the definition for the 
> > > registration from, but I would be more than happy is someone could 
> > > point me to this information.
> > >
> > > Considering existing schemata for LSRegistration, I found this one:
> > > http://anonsvn.internet2.edu/svn/perfsonar/trunk/geant2_java-xml-ls/doc/schema/rnc/LSRegisterRequest.rnc 
> > >
> > >
> > > What is more or less exactly, what our service is registering at the 
> > > moment.
> > > I hope you understand now my problems.
> > >   
> >
> >
> > As I stated before, I am not an expert on what documents went into 
> > the perfSONAR 3.x release series, but I am positive that something 
> > exists.  If someone (Maciej/Szymon/Nicolas/Loukik) knows where these 
> > are, could they please let Verena and the others know?
> >
> > -jason
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > <?xml version='1.0' encoding='UTF-8'?>
> > <nmwg:message type="LSRegisterRequest"
> >               id="msg1"
> >               
> > xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";
> >               xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/";
> >               
> > xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/"; 
> >
> >               xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";
> >               xmlns:iperf= "http://ggf.org/ns/nmwg/tools/iperf/2.0/";>
> >
> >   <nmwg:metadata id="serviceLookupInfo">
> >     <perfsonar:subject id="commonParameters" 
> > xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";>
> >       <psservice:service id="serviceParameters" 
> > xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/";> 
> >
> >         <psservice:serviceName>Internet2 perfSONAR-BUOY 
> > MA</psservice:serviceName>
> >         
> > <psservice:accessPoint>http://dc211.internet2.edu:8099/perfSONAR_PS/services/perfSONARBUOY</psservice:accessPoint> 
> >
> >         <psservice:serviceType>MA</psservice:serviceType>
> >         <psservice:serviceDescription>Washington DC perfSONAR-BUOY 
> > MA</psservice:serviceDescription>
> >       </psservice:service>
> >     </perfsonar:subject>
> >   </nmwg:metadata>
> >       <nmwg:data id="data-1" metadataIdRef="serviceLookupInfo">
> >     <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"; 
> > id="metadata-1">
> >       <iperf:subject 
> > xmlns:iperf="http://ggf.org/ns/nmwg/tools/iperf/2.0/"; id="subject-1">
> >         <nmwgt:endPointPair 
> > xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
> >           <nmwgt:src type="ipv4" value="207.75.165.145" port="5434" />
> >           <nmwgt:dst type="ipv4" value="192.52.179.221" port="5434" />
> >         </nmwgt:endPointPair>
> >       </iperf:subject>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/tools/iperf/2.0</nmwg:eventType>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/characteristics/bandwidth/acheiveable/2.0</nmwg:eventType> 
> >
> >       <nmwg:parameters id="parameters-1">
> >         <nmwg:parameter name="windowSize">1m</nmwg:parameter>
> >         <nmwg:parameter name="timeDuration">30</nmwg:parameter>
> >         <nmwg:parameter name="interval">2</nmwg:parameter>
> >         <nmwg:parameter name="protocol">TCP</nmwg:parameter>
> >       </nmwg:parameters>
> >     </nmwg:metadata>
> >   </nmwg:data>
> >  
> >   <nmwg:data id="data-3" metadataIdRef="serviceLookupInfo">
> >     <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"; 
> > id="metadata-3">
> >       <iperf:subject 
> > xmlns:iperf="http://ggf.org/ns/nmwg/tools/iperf/2.0/"; id="subject-3">
> >         <nmwgt:endPointPair 
> > xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
> >           <nmwgt:src type="ipv4" value="192.52.179.221" port="5434" />
> >           <nmwgt:dst type="ipv4" value="207.75.165.145" port="5434" />
> >         </nmwgt:endPointPair>
> >       </iperf:subject>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/tools/iperf/2.0</nmwg:eventType>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/characteristics/bandwidth/acheiveable/2.0</nmwg:eventType> 
> >
> >       <nmwg:parameters id="parameters-3">
> >         <nmwg:parameter name="windowSize">1m</nmwg:parameter>
> >         <nmwg:parameter name="timeDuration">30</nmwg:parameter>
> >         <nmwg:parameter name="interval">2</nmwg:parameter>
> >         <nmwg:parameter name="protocol">TCP</nmwg:parameter>
> >       </nmwg:parameters>
> >     </nmwg:metadata>   </nmwg:data>   </nmwg:message>
> >   
> > ------------------------------------------------------------------------
> >
> > <?xml version='1.0' encoding='UTF-8'?>
> > <nmwg:message type="LSRegisterRequest"
> >               id="msg1"
> >               
> > xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";
> >               xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/";
> >               
> > xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/"; 
> >
> >               xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";
> >               
> > xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";
> >               xmlns:snmp="http://ggf.org/ns/nmwg/tools/snmp/2.0/";>
> >
> >   <nmwg:metadata id="serviceLookupInfo">
> >     <perfsonar:subject id="commonParameters" 
> > xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";>
> >       <psservice:service id="serviceParameters" 
> > xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/";> 
> >
> >         <psservice:serviceName>Internet2 SNMP MA</psservice:serviceName>
> >         
> > <psservice:accessPoint>http://dc211.internet2.edu:8083/perfSONAR_PS/services/snmpMA</psservice:accessPoint> 
> >
> >         <psservice:serviceType>MA</psservice:serviceType>
> >         <psservice:serviceDescription>Washington DC SNMP 
> > MA</psservice:serviceDescription>
> >       </psservice:service>
> >     </perfsonar:subject>
> >   </nmwg:metadata>
> >       <nmwg:data id="data1" metadataIdRef="serviceLookupInfo">
> >     <nmwg:metadata id="meta1">
> >       <netutil:subject id="subj1" 
> > xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
> >         <nmwgt:interface 
> > xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
> >           <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
> >           <nmwgt:ifName>eth0</nmwgt:ifName>
> >           <nmwgt:ifDescription>Interface eth0 on dc211 
> > (external)</nmwgt:ifDescription>
> >           <nmwgt:ifAddress type="ipv4">192.52.179.211</nmwgt:ifAddress>
> >           <nmwgt:direction>in</nmwgt:direction>
> >           <nmwgt:capacity>100000000</nmwgt:capacity>
> >         </nmwgt:interface>
> >       </netutil:subject>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType> 
> >
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
> >       <nmwg:parameters id="param1">
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> 
> >
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter> 
> >
> >       </nmwg:parameters>
> >     </nmwg:metadata>
> >   </nmwg:data>
> >
> >   <nmwg:data id="data2" metadataIdRef="serviceLookupInfo">
> >     <nmwg:metadata id="meta2">
> >       <netutil:subject id="subj2" 
> > xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
> >         <nmwgt:interface 
> > xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
> >           <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
> >           <nmwgt:ifName>eth0</nmwgt:ifName>
> >           <nmwgt:ifDescription>Interface eth0 on dc211 
> > (external)</nmwgt:ifDescription>
> >           <nmwgt:ifAddress type="ipv4">192.52.179.211</nmwgt:ifAddress>
> >           <nmwgt:direction>out</nmwgt:direction>
> >           <nmwgt:capacity>100000000</nmwgt:capacity>
> >         </nmwgt:interface>
> >       </netutil:subject>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType> 
> >
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
> >       <nmwg:parameters id="param2">
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> 
> >
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter> 
> >
> >       </nmwg:parameters>
> >     </nmwg:metadata>
> >   </nmwg:data>
> >
> >   <nmwg:data id="data3" metadataIdRef="serviceLookupInfo">
> >     <nmwg:metadata id="meta3">
> >       <netutil:subject id="subj3" 
> > xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
> >         <nmwgt:interface 
> > xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
> >           <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
> >           <nmwgt:ifName>eth1</nmwgt:ifName>
> >           <nmwgt:ifDescription>Interface eth0 on dc211 
> > (internal)</nmwgt:ifDescription>
> >           <nmwgt:ifAddress type="ipv4">10.0.0.2</nmwgt:ifAddress>
> >           <nmwgt:direction>in</nmwgt:direction>
> >           <nmwgt:capacity>100000000</nmwgt:capacity>
> >         </nmwgt:interface>
> >       </netutil:subject>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType> 
> >
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
> >       <nmwg:parameters id="param3">
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> 
> >
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter> 
> >
> >       </nmwg:parameters>
> >     </nmwg:metadata>
> >
> >     <nmwg:metadata id="meta4">
> >       <netutil:subject id="subj4" 
> > xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
> >         <nmwgt:interface 
> > xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
> >           <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
> >           <nmwgt:ifName>eth1</nmwgt:ifName>
> >           <nmwgt:ifDescription>Interface eth0 on dc211 
> > (internal)</nmwgt:ifDescription>
> >           <nmwgt:ifAddress type="ipv4">10.0.0.2</nmwgt:ifAddress>
> >           <nmwgt:direction>in</nmwgt:direction>
> >           <nmwgt:capacity>100000000</nmwgt:capacity>
> >         </nmwgt:interface>
> >       </netutil:subject>
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType> 
> >
> >       
> > <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
> >       <nmwg:parameters id="param4">
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> 
> >
> >         <nmwg:parameter 
> > name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter> 
> >
> >       </nmwg:parameters>
> >     </nmwg:metadata>
> >   </nmwg:data>
> >  
> > </nmwg:message>
> >