perfsonar development work

[Jason Zurawski <>]

Verena;

> > > > > We are discussing what information services should register into LS,
> > > > > besides service URL.  Recently, Jason advised to recognize service
> > > > > type by eventtypes, rather than by arbitrary strings like one found
> > > > > in "servicetype" parameter.
> > > > >
> > > > > This seems to work fine with RRD MA services, but not all other
> > > > > services submit information about eventtypes when registering to LS.
> > > > > Is there a guidance what schema should be supported in this case?
> > > > >           
> > > > Could you be more specific? Which services are not registering
> > > > eventTypes? This disrupts the dLS (and gLS) summarization model as
> > > > well...
> > > >         
> > > So far I am aware of flowsa MA, developed at Surfnet ,  and  BWCTL and
> > > Hades by Win-Labor.  Verena is going to add support for eventtypes,
> > > but was asking of the syntax to be supported. Could you please advise
> > > her?
> > >       
> > I am not sure I understand the request, services should be registering
> > with the LS in a manner similar to the RRD MA etc. where they register a
> > Service metadata description, and the metadata elements they are
> > personally responsible for.  These metadata elements *should* include
> > eventType elements for the particular data they pertain to.  The
> > examples that can be examined are related to the RRD MA (this is the
> > only one I personally know of, perhaps there are more).
> >
> >     
> Good to know. In the past, I undertook several attempts, to implement 
> something based on an example of another service. However, this did not work 
> in a lot of cases, as examples tend to be outdated sooner or later.
>
> It would be a good idea, if anyone expects services to implement based on an 
> example, to state this clearly which example, and if this is going to be the 
> one which is specified (hopefully) sooner or later.
>   


I am attaching two examples from perfSONAR-PS services (SNMP MA and 
perfSONAR-BUOY).  These metadata elements are taken directly from the 
store.xml files for each service respectively (making LS registration 
trivial).  If your service is not using eventTypes currently, I would 
suggest placing them in your storage, or injecting them into the 
LSRegistration message.


> > > > I think we should concentrate effort to fix this on the service
> > > > registration side, not by adding heuristics on the client side (which
> > > > would then need to be copied to every other client).
> > > >         
> > > Yes, sure - we've already introduced some perfsonarUI code that relies
> > > on eventtypes registered in LS and are trying to figure out how it
> > > will work with services other than RRD MA.
> > >       
> > I can't really stress this enough: discovery will not work unless
> > services are registering in the same manner.  
> >     
>    ^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> But that's exactly the point! All we want to know, is HOW this should look 
> like.  We cannot implement it in the same manner, if we do not know this.
>   


Perhaps I am missing something, but isn't there documents provided in 
the XML LS that describe what is expected?  The general idea is to 
include a service description, and the metadata they manage.  It would 
be impossible to enumerate what every single service must provide, but 
the basic idea is to include measurement metadata containing at a 
minimum a subject with a topological element as well as eventType(s).  
This is what the various perfSONAR-PS services provide, and the RRD-MA 
does as well. 


> > If a service does not make 
> > available to the LS:
> >
> > a) Address information (ipv4 or perhaps ipv6)
> > d) Domain information (fully qual hostname will do)
> > c) any and all possible eventTypes
> >
> > Discovery will be meaningless.  If service authors have questions or
> > concerns about this, lets start talking about it very soon, and if you
> > haven't done so *please* read the LS API document, this should explain
> > everything:
> >
> > http://anonsvn.internet2.edu/svn/nmwg/trunk/nmwg/doc/dLS/gLS/phase_1_color.
> > html
> >
> >     
> Good to know this one also. It is not obvious to me (and considering that 
> quite a lot of other services have not implemented this, it seems to be not 
> obvious to others, too) that the API for dLS is relevant for implementing the 
> API to LS! Sorry, but how do you expect me and others to find exactly this 
> document if noone points out that it is the one you need to implement when 
> properly registering to LS?
>
> By the way, I had a look into it. Unfortunately it does not point to a schema 
> definition for the elements to be registered with a normal LS. So, still, I 
> do not know, how I should do this exactly.
>
> It's not that I need a discussion what elements should be registered. I just 
> need to know which elements you need, and in what format I should register 
> them, that's all. 
>   


I think you are confusing two issues here: what is necessary for 
services to register to LS instances and what LS instances need to do to 
distribute this information via the gLS mechanism.  The gLS document I 
pasted above does not deal with the former and I do believe descriptions 
exist for this task exist already and Maciej should be able to point 
people to the exact locations.

The requirements for services registering to the LS have not changed 
since the original LS was released some time ago.  We have always asked 
that each service provide a service information description as well as 
metadata that they manage.  In the provided metadata it is assumed that 
eventType(s) as well as the necessary domain and address information is 
provided (each time I helped design new metadata descriptions for the 
various services we ensured that this information was available).

The gLS requirements (and the API specification in the above document) 
rely on the same sane registration practices, and should *not* require 
additional service modifications to function properly (we designed it 
this way on purpose to minimize the impact on service designers in the 
middle of release cycles).  If a service is not registering properly, 
this will damage discovery and the goal of distributing information.


> To summarize: I have absolutly no concerns with the content of a registration 
> message.  All I want to know is the proper definition of this.
>
> I'm not sure, where the RRDMA got the definition for the registration from, 
> but I would be more than happy is someone could point me to this information.
>
> Considering existing schemata for LSRegistration, I found this one:
> http://anonsvn.internet2.edu/svn/perfsonar/trunk/geant2_java-xml-ls/doc/schema/rnc/LSRegisterRequest.rnc
>
> What is more or less exactly, what our service is registering at the moment.
> I hope you understand now my problems.
>   


As I stated before, I am not an expert on what documents went into the 
perfSONAR 3.x release series, but I am positive that something exists.  
If someone (Maciej/Szymon/Nicolas/Loukik) knows where these are, could 
they please let Verena and the others know?

-jason



<?xml version='1.0' encoding='UTF-8'?>
<nmwg:message type="LSRegisterRequest"
              id="msg1"
              xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";
              xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/";
              xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/";
              xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";
              xmlns:iperf= "http://ggf.org/ns/nmwg/tools/iperf/2.0/";>

  <nmwg:metadata id="serviceLookupInfo">
    <perfsonar:subject id="commonParameters" xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";>
      <psservice:service id="serviceParameters" xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/";>
        <psservice:serviceName>Internet2 perfSONAR-BUOY MA</psservice:serviceName>
        <psservice:accessPoint>http://dc211.internet2.edu:8099/perfSONAR_PS/services/perfSONARBUOY</psservice:accessPoint>
        <psservice:serviceType>MA</psservice:serviceType>
        <psservice:serviceDescription>Washington DC perfSONAR-BUOY MA</psservice:serviceDescription>
      </psservice:service>
    </perfsonar:subject>
  </nmwg:metadata>
    
  <nmwg:data id="data-1" metadataIdRef="serviceLookupInfo">
    <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"; id="metadata-1">
      <iperf:subject xmlns:iperf="http://ggf.org/ns/nmwg/tools/iperf/2.0/"; id="subject-1">
        <nmwgt:endPointPair xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
          <nmwgt:src type="ipv4" value="207.75.165.145" port="5434" />
          <nmwgt:dst type="ipv4" value="192.52.179.221" port="5434" />
        </nmwgt:endPointPair>
      </iperf:subject>
      <nmwg:eventType>http://ggf.org/ns/nmwg/tools/iperf/2.0</nmwg:eventType>
      <nmwg:eventType>http://ggf.org/ns/nmwg/characteristics/bandwidth/acheiveable/2.0</nmwg:eventType>
      <nmwg:parameters id="parameters-1">
        <nmwg:parameter name="windowSize">1m</nmwg:parameter>
        <nmwg:parameter name="timeDuration">30</nmwg:parameter>
        <nmwg:parameter name="interval">2</nmwg:parameter>
        <nmwg:parameter name="protocol">TCP</nmwg:parameter>
      </nmwg:parameters>
    </nmwg:metadata>
  </nmwg:data>
 
  <nmwg:data id="data-3" metadataIdRef="serviceLookupInfo">
    <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"; id="metadata-3">
      <iperf:subject xmlns:iperf="http://ggf.org/ns/nmwg/tools/iperf/2.0/"; id="subject-3">
        <nmwgt:endPointPair xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
          <nmwgt:src type="ipv4" value="192.52.179.221" port="5434" />
          <nmwgt:dst type="ipv4" value="207.75.165.145" port="5434" />
        </nmwgt:endPointPair>
      </iperf:subject>
      <nmwg:eventType>http://ggf.org/ns/nmwg/tools/iperf/2.0</nmwg:eventType>
      <nmwg:eventType>http://ggf.org/ns/nmwg/characteristics/bandwidth/acheiveable/2.0</nmwg:eventType>
      <nmwg:parameters id="parameters-3">
        <nmwg:parameter name="windowSize">1m</nmwg:parameter>
        <nmwg:parameter name="timeDuration">30</nmwg:parameter>
        <nmwg:parameter name="interval">2</nmwg:parameter>
        <nmwg:parameter name="protocol">TCP</nmwg:parameter>
      </nmwg:parameters>
    </nmwg:metadata> 
  </nmwg:data> 
  
</nmwg:message>

<?xml version='1.0' encoding='UTF-8'?>
<nmwg:message type="LSRegisterRequest"
              id="msg1"
              xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";
              xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/";
              xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/";
              xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";
              xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";
              xmlns:snmp="http://ggf.org/ns/nmwg/tools/snmp/2.0/";>

  <nmwg:metadata id="serviceLookupInfo">
    <perfsonar:subject id="commonParameters" xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/";>
      <psservice:service id="serviceParameters" xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/";>
        <psservice:serviceName>Internet2 SNMP MA</psservice:serviceName>
        <psservice:accessPoint>http://dc211.internet2.edu:8083/perfSONAR_PS/services/snmpMA</psservice:accessPoint>
        <psservice:serviceType>MA</psservice:serviceType>
        <psservice:serviceDescription>Washington DC SNMP MA</psservice:serviceDescription>
      </psservice:service>
    </perfsonar:subject>
  </nmwg:metadata>
    
  <nmwg:data id="data1" metadataIdRef="serviceLookupInfo">
    <nmwg:metadata id="meta1">
      <netutil:subject id="subj1" xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
        <nmwgt:interface xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
          <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
          <nmwgt:ifName>eth0</nmwgt:ifName>
          <nmwgt:ifDescription>Interface eth0 on dc211 (external)</nmwgt:ifDescription>
          <nmwgt:ifAddress type="ipv4">192.52.179.211</nmwgt:ifAddress>
          <nmwgt:direction>in</nmwgt:direction>
          <nmwgt:capacity>100000000</nmwgt:capacity>
        </nmwgt:interface>
      </netutil:subject>
      <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType>
      <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
      <nmwg:parameters id="param1">
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter>
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter>
      </nmwg:parameters>
    </nmwg:metadata>
  </nmwg:data>

  <nmwg:data id="data2" metadataIdRef="serviceLookupInfo">
    <nmwg:metadata id="meta2">
      <netutil:subject id="subj2" xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
        <nmwgt:interface xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
          <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
          <nmwgt:ifName>eth0</nmwgt:ifName>
          <nmwgt:ifDescription>Interface eth0 on dc211 (external)</nmwgt:ifDescription>
          <nmwgt:ifAddress type="ipv4">192.52.179.211</nmwgt:ifAddress>
          <nmwgt:direction>out</nmwgt:direction>
          <nmwgt:capacity>100000000</nmwgt:capacity>
        </nmwgt:interface>
      </netutil:subject>
      <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType>
      <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
      <nmwg:parameters id="param2">
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter>
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter>
      </nmwg:parameters>
    </nmwg:metadata>
  </nmwg:data>

  <nmwg:data id="data3" metadataIdRef="serviceLookupInfo">
    <nmwg:metadata id="meta3">
      <netutil:subject id="subj3" xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
        <nmwgt:interface xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
          <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
          <nmwgt:ifName>eth1</nmwgt:ifName>
          <nmwgt:ifDescription>Interface eth0 on dc211 (internal)</nmwgt:ifDescription>
          <nmwgt:ifAddress type="ipv4">10.0.0.2</nmwgt:ifAddress>
          <nmwgt:direction>in</nmwgt:direction>
          <nmwgt:capacity>100000000</nmwgt:capacity>
        </nmwgt:interface>
      </netutil:subject>
      <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType>
      <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
      <nmwg:parameters id="param3">
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter>
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter>
      </nmwg:parameters>
    </nmwg:metadata>

    <nmwg:metadata id="meta4">
      <netutil:subject id="subj4" xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/";>
        <nmwgt:interface xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";>
          <nmwgt:hostName>dc211.internet2.edu</nmwgt:hostName>
          <nmwgt:ifName>eth1</nmwgt:ifName>
          <nmwgt:ifDescription>Interface eth0 on dc211 (internal)</nmwgt:ifDescription>
          <nmwgt:ifAddress type="ipv4">10.0.0.2</nmwgt:ifAddress>
          <nmwgt:direction>in</nmwgt:direction>
          <nmwgt:capacity>100000000</nmwgt:capacity>
        </nmwgt:interface>
      </netutil:subject>
      <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType>
      <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType>
      <nmwg:parameters id="param4">
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter>
        <nmwg:parameter name="supportedEventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter>
      </nmwg:parameters>
    </nmwg:metadata>
  </nmwg:data>
 
</nmwg:message>