perfsonar development work

[Verena Venus <>]

Hi Jason, all,

Am Mittwoch, 7. Mai 2008 16:38:13 schrieb Jason Zurawski:
> Jochen & All;
>
> > I'm mainly responsible for the rsync and Perl Storable part of the story,
> > that is and will not be part of perfSONAR. For the perfSONAR specific
> > stuff others are more competent and should answer the questions.
>
>  From a protocol standpoint, I am really interested in seeing what
> messages will be used to actually convey the message that an 'out of
> band transfer' is necessary.  Even though the focus is currently on the
> behind the scenes work, there needs to be some message kicking off this
> action that someone will need to formulate.
>
> If you haven't thought much about what the message looks like yet, I
> would suggest something similar to this:
>
> <nmwg:message type="SetupDataRequest"
> xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/";>
>
>   <nmwg:metadata id="m-1">
>     <!-- whatever it looks like -->
>   </nmwg:metadata>
>
>   <nmwg:data id="d-1" metadataIdRef="m-1">
>     <nmwg:key id="k-1">
>       <nmwg:parameters id="pk-1">
>         <nmwg:parameter name="rsync">something...</nmwg:parameter>
>       </nmwg:parameters>
>     </nmwg:key>
>   </nmwg:data>
>
> </nmwg:message>
>
>
> Even though this is a Hades only extension for now, properly specifying
> what modifications are to be made is pretty important.  Especially for
> other MA developers who are Jaded on XML parsing.
>
Well, I have thought of this, and of course we want to find a solution which 
is useful for all perfSONAR services. So, I tried to discuss this with 
Martin, but he seems to be unavailable...
That's the reason why I did not say anything to the list, it really is just a 
mere idea, far from being "ready" or "fixed". 

We do not intend to implement something which might spoil perfSONAR and then 
leave you with the facts. So, concerning the perfSONAR part, all the details 
you want to know aren't there yet. And concerning the actual data format we 
use between CNM and Hades, this does not interfere with perfSONAR as we have 
it now. And it is not intended to be interfering, in the contrary. It is 
intended to be used in addition, if the need for it is there. 

All we want to achieve is what was roughly discussed in Zagreb. And the 
perfSONAR relevant thing has to be done still. So, I wanted to work out a 
sensible solution first and then bring it to the list, before discussing 
vague ideas. But if this thread will lead to a good solution, I'm happy with 
that :)  

What I had in mind is somewhat similar to your example:

<nmwg:message xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/";
             xmlns:nmwgm="http://ggf.org/ns/nmwg/time/2.0/";
             xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";
             xmlns:select="http://ggf.org/ns/nmwg/ops/select/2.0/";
             xmlns:hades="http://ggf.org/ns/nmwg/tools/hades/";
             id="1208356112" type="MetadataKeyRequest">
  <nmwg:metadata id="keymeta">
    <key:subject id="keysubject"/>
    <key:parameters id ="keyparam1">
      <nmwg:parameter name="transfer" value="bulk"/>
      <nmwg:parameter name="format" value="perl-storable"/>
      <nmwg:parameter name="validity" value="30000"/> #seconds?
    </key:parameters>
  </nmwg:metadata>
  <nmwg:metadata id="meta1" metadataIdRef="keymeta">
    <hades:subject id="subject1">
      <nmwgt:endPointPair>
        <nmwgt:src type="IFname" value="Ljubljana_ARNES"/>
        <nmwgt:dst type="IFname" value="Amsterdam_SURFnet"/>
      </nmwgt:endPointPair>
    </hades:subject>
    <nmwg:eventType>http://ggf.org/ns/nmwg/tools/hades/</nmwg:eventType>
  </nmwg:metadata>
  <nmwg:metadata id="meta2" metadataIdRef="meta1">
    <select:subject id="subject2"/>
    <select:parameters id="param1">
      <nmwg:parameter name="startTime">1208296800</nmwg:parameter>
      <nmwg:parameter name="endTime">1208383200</nmwg:parameter>
    </select:parameters>
    <nmwg:eventType>http://ggf.org/ns/nmwg/ops/select/</nmwg:eventType>
  </nmwg:metadata>
  <nmwg:data id="data1" metadataIdRef="meta2"/>
</nmwg:message>

So, this seems to be more or less the same direction.  The sooner we get this 
specified, the better. 

Honestly, I don't see, where the problem is. We never said that we want to 
actually _change_ something in perfSONAR. From client point of view, nothing 
happens. You only get an _additional_ possibility to get large chunks of data 
using perfSONAR as control channel. You don't have to use it, if it doesn't 
fit your needs, you can still request data as hitherto. And it does not mean, 
that we do not like to participate in an overall solution to transfer larger 
data chunks. But that is not there at the moment. When it will be there, we 
will of course implement it. But I don't see the point why this should affect 
the transfer mechanism between CNM and Hades, we are using now. Moreover, I 
think we can learn quite a lot from this example, which will benefit a 
general solution.

However, comments on the above example are welcome :)

Verena


> -jason
>
> > regards,
> > Jochen
> >
> > On Tuesday 06 May 2008 16:07, Jeff W. Boote wrote:
> >> On May 6, 2008, at 3:37 AM, Jochen Reinwand wrote:
> >>> On Monday 05 May 2008 17:46, Joe Metzger wrote:
> >>>> Jeff & Jochen,
> >>>> I am not too concerned about the technical details about how
> >>>> to use messages to exchange RSYC URLs.  I am concerned
> >>>> about the big picture and how this decision affects the
> >>>> future of the perfSONAR protocols.
> >>>
> >>> The only thing that is important for the perfSONAR protocol is the
> >>> way the
> >>> rsync URLs are put into the perfSONAR messages. What this URLs mean
> >>> is out of
> >>> scope for perfSONAR. This is the basic idea: Make your data
> >>> available via
> >>> some (most likely otherwise standardised) mechanism and use the
> >>> power of
> >>> perfSONAR, especially regarding meta data, to point the other
> >>> communication
> >>> partners to that source.
> >>
> >> To a large extent I agree with you.
> >>
> >> The 'however' is that I think we need to come up with a language
> >> independent perfSONAR based efficient data transport mechanism.
> >> Otherwise, other services will create something similar to what you
> >> are doing here and over time it could become very difficult to write
> >> client applications that can consume multiple sources of data.
> >> (Consider: how easy will it be for pS-UI to consume data using this
> >> mechanism given the perl objects?) Given that the over-arching goal of
> >> perfSONAR is to make many sources of data available in a common
> >> infrastructure, surely you can see how this language specific
> >> transport is at least somewhat counter to that goal.
> >>
> >> But, I do see that this trade-off is needed for you in the short term.
> >> This is why I would like to hear more details on how much of the issue
> >> is with parsing, and how much has to do with in-lining the data with
> >> the control communication. So we can attempt to rectify that situation
> >> in the medium to long term.
> >>
> >> The other concern I have is simply that I would like to see
> >> specifically how you intend to put the URL's into the messages, and I
> >> guess I would like to see the eventType clearly indicate that this
> >> type of data transport is outside the scope of perfSONAR.
> >>
> >> Each new development group that comes on board looks at existing
> >> services for the example on how to write services. This particular
> >> example trades interoperability for performance. This is reasonable
> >> given your goals, but we need to clearly explain the limits of this
> >> type of transport to ensure this method is not copied by those that
> >> don't understand the limitations.
> >>
> >> jeff
> >>
> >>>> Can you please clarify for me if this will be a private,
> >>>> internal interface for your service, or if you expect this to be a
> >>>> change
> >>>> to the public perfSONAR protocol that will be used by GUI's and
> >>>> other services to access the data stored in your service?
> >>>
> >>> It's an enhancement to the protocol and another way of accessing
> >>> Hades (and
> >>> perhaps even other) data. The first one is a simple enhancement
> >>> possibly
> >>> useful in some other cases. The later is a new functionality that
> >>> has nothing
> >>> to do with perfSONAR, but that we should not hide from users either.
> >>>
> >>>> What I hope you are doing is using a perfSONAR-like protocol internal
> >>>> to a perfSONAR service to move data around.  If this is what
> >>>> is going on, then great, I have no problem. Developers must be able
> >>>> to architect the internals of their services in any way that
> >>>> meets their requirements, and passing around language specific
> >>>> serialized objects is expected internal to a service.
> >>>
> >>> Perhaps you like this description better: It is a perfSONAR
> >>> compliant service
> >>> enhancement that allows fast data access via Perl storables and
> >>> rsync. It is
> >>> especially useful for a specific purpose (fast interaction between to
> >>> specific perfSONAR tools), but can be useful in other cases, too.
> >>>
> >>> Can you please give a good reason why it shouldn't be possible to
> >>> include a
> >>> URL as data in perfSONAR messages? I simply don't see the point...
> >>>
> >>> Jochen
> >>>
> >>> --
> >>> Jochen Reinwand, DFN-Labor
> >>> Friedrich-Alexander-Universität Erlangen-Nürnberg
> >>> Regionales RechenZentrum Erlangen (RRZE)
> >>> Martensstraße 1, 91058 Erlangen, Germany
> >>> Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941
> >>> 
> >>> www.win-labor.dfn.de



-- 
Verena Venus, DFN-Labor
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales RechenZentrum Erlangen (RRZE)
Martensstraße 1, 91058 Erlangen, Germany
Tel. +49 9131 85-28738, -28800, Fax +49 9131 302941


www.win-labor.dfn.de