perfsonar development work

[Jason Zurawski <>]

Slawomir;

> But you wanted to implement Discovery-request to make independent of
> XQuery. 


The discovery message is not a full replacement for the XQuery 
interface, it addresses a different need (see the design document).

Because the LS features a query language and storage format that are so 
closely tied, a binding is created between implementations where this is 
not the case throughout the rest of the framework.  For example, anyone 
could implement a service similar to the RRDMA that does not use a 
'store.xml' file as a way to describe metadata - as long as the queries 
are recognized on the front side, the backend organization will not matter.

Native queries to the LS are allowed to 'pass through' unaltered, 
because of this it is important that the backend storage be
_as close as physically possible_ to ensure that the behavior between 
the services is as close as it can be.  To paraphrase something Aaron 
said recently, this is an example of the "Principle of least 
astonishment" 
(http://en.wikipedia.org/wiki/Principle_of_least_astonishment), the user 
should expect to be able to send the same query (within reason - see 
below why this is not always possible) and have it behave in the same 
manner when dealing with an hLS, regardless of who implemented it.

A reason why I went into a lot of detail in the gLS design document 
regarding the storage format is because the query interface directly 
exposes this.  Client/API developers only have this document to look at 
currently, so it is very important that we both follow the same design 
specifications and not introduce anomalous behavior.


> Full conformity with XQuery with difference dbs is impossible


I do not agree, this is not an XQuery languague issue - this is a data 
organization issue.  It may be true that DBXML does not implement the 
same version of the XQuery/XPath standards as eXist.  In this particular 
case the sent query did not fail because the syntax was different 
between the two versions, it 'failed' (did not return information) 
because data was not in the expected location.

-jason


> On Tue, 2009-08-11 at 09:56 +0000, Jason Zurawski wrote:
> > Slawomir;
> >
> > Why?  
> >
> > This was specified in the original design document and has a severe impact on 
> > interoperability.  If a user can't send the same query to either 
> > implementation and have it work, something is broken and needs to be fixed 
> > immediately.
> >
> > -jason
> >
> > -----Original Message-----
> > From: Slawomir Trzaszczka 
> > <>
> >
> > Date: Tue, 11 Aug 2009 10:54:26 
> > To: <>
> > Cc: Szymon 
> > Trocha<>;
> >  
> > <>;
> >  
> > <>
> > Subject: Re: [pS-dev] Lookup service playground tab
> >
> >
> > Hi Jason,
> >
> > Your query is incorrect. Try :
> >
> > declare namespace nmwg="http://ggf.org/ns/nmwg/base/2.0/";;
> > /nmwg:store[@type="LSStore-control"]/nmwg:metadata
> >
> > I changed value of the parameter 'type' from 'LSStore-summary' to
> > 'LSStrore-control'
> >
> > Regards,
> >
> > Slawek
> >
> >
> > On Mon, 2009-08-10 at 07:51 -0400, Jason Zurawski wrote:
> > > I have explained this on numerous occasions before, and nothing has 
> > > really changed: we are aware that the IP summarization is not as tight 
> > > as some would like them to be.  There is ongoing work with a student at 
> > > the University of Delaware to improve the algorithm and he has a report 
> > > on his results so far available here:
> > >
> > > http://code.google.com/p/perfsonar-ps/wiki/IPSummarization
> > >
> > > and
> > >
> > > https://damsl.cis.udel.edu/svn/perl-iptrie/README.html
> > >
> > > I have not had a chance to incorporate his work into the gLS releases 
> > > yet, so this is still untested in a production environment.  We will be 
> > > working towards releasing this before the end of the year.  Until then 
> > > our suggestion of how to work around this still stands: multiple queries 
> > > to the result hLSs will be be required to determined if these instances 
> > > do or do not hold what you are looking for.
> > >
> > > As for your results below I took a look at the hLS installed on 
> > > p-mdm.ps-lhcopn.fnal.gov, I would encourage all to see the internal view 
> > > here:
> > >
> > > http://dc211.internet2.edu/cgi-bin/pA/view.cgi?hls=http://p-mdm.ps-lhcopn.fnal.gov:8080/geant2-java-xml-ls/services/LookupService
> > >
> > > This is an MDM hLS, so I am not aware of the specifics of how it works 
> > > on the inside.  According to the view of the hLS summary set 
> > > (represented as "Collection: 
> > > http://ogf.org/ns/nmwg/tools/org/perfsonar/service/lookup/discovery/xquery/2.0"; 
> > > and "Store Type: LSStore-summary" - scroll all the way to the bottom) 
> > > there is nothing there.  This may mean:
> > >
> > > 1) I am not querying this set correctly, I will let the MDM hLS 
> > > maintainer answer if I am doing this correctly.  If there is a problem 
> > > with my query, that would indicate a bug in my CGIs which I will gladly 
> > > fix.  I am sending this query:
> > >
> > > <nmwg:message type="LSQueryRequest"
> > >                id="LSQueryRequest"
> > >                xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/";
> > >  
> > > xmlns:xquery="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/lookup/xquery/1.0/";> 
> > >
> > >    <nmwg:metadata id="meta1">
> > >      <xquery:subject id="sub1">
> > >    declare namespace nmwg="http://ggf.org/ns/nmwg/base/2.0/";;
> > >    
> > > /nmwg:store[@type="LSStore-summary"]/nmwg:metadata
> > >      </xquery:subject>
> > >  
> > > <nmwg:eventType>http://ogf.org/ns/nmwg/tools/org/perfsonar/service/lookup/discovery/xquery/2.0</nmwg:eventType> 
> > >
> > >    </nmwg:metadata>
> > >    <nmwg:data metadataIdRef="meta1" id="d1"/>
> > > </nmwg:message>
> > >
> > > I do get a response (but it is vaugue):
> > >
> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <soapenv:Envelope 
> > > xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; 
> > > xmlns:xsd="http://www.w3.org/2001/XMLSchema"; 
> > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";><soapenv:Body><nmwg:message 
> > > xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"; id="LSQueryRequest_resp" 
> > > messageIdRef="LSQueryRequest" type="LSQueryResponse"><nmwg:metadata 
> > > id="LSQueryResponseMetadata"><nmwg:eventType>success.ls.query</nmwg:eventType></nmwg:metadata><nmwg:data 
> > > id="LSQueryResponseData" 
> > > metadataIdRef="LSQueryResponseMetadata"><psservice:datum 
> > > xmlns:psservice="http://ggf.org/ns/nmwg/tools/org/perfsonar/service/1.0/"/></nmwg:data></nmwg:message></soapenv:Body></soapenv:Envelope>
> > >
> > >
> > >
> > > 2) The set is empty and the service is not summarizing, would indicate a 
> > > bug in the service
> > >
> > > 3) The service just came up and did not have a chance to summarize yet 
> > > (unlikely, this appears to have been registered for a while but I can't 
> > > prove this from where I sit)
> > >
> > > -jason
> > >
> > >
> > > > First try:
> > > > 40963 ms
> > > > Query
> > > > urn:ogf:network:node=192.16.166.26
> > > > Services
> > > >         http://216.255.240.2:8085/perfSONAR_PS/services/pSB
> > > >         http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA
> > > >         http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/snmpMA
> > > >         http://ps3.es.net:8080/perfSONAR_PS/services/snmpMA
> > > >         http://frown.es.net:8085/perfSONAR_PS/services/pSB
> > > >         http://perfsonar-1.t2.ucsd.edu:8085/perfSONAR_PS/services/pSB
> > > >
> > > > Second try:
> > > > 41670 ms
> > > > Query
> > > > urn:ogf:network:node=192.16.166.26
> > > > Services
> > > >         http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/IUsnmpMA
> > > >         http://rrdma.net.internet2.edu:8080/perfSONAR_PS/services/snmpMA
> > > >         http://frown.es.net:8085/perfSONAR_PS/services/pSB
> > > >         http://perfsonar-1.t2.ucsd.edu:8085/perfSONAR_PS/services/pSB
> > > >         http://ps3.es.net:8080/perfSONAR_PS/services/snmpMA
> > > >         http://216.255.240.2:8085/perfSONAR_PS/services/pSB
> > > >
> > > > Third try:
> > > > 433 ms
> > > > Query
> > > > urn:ogf:network:node=192.16.166.26
> > > > Services
> > > >
> > > > Fourth try:
> > > > 3518 ms
> > > > Query
> > > > urn:ogf:network:node=192.16.166.26
> > > > Services
> > > >
> > > >
> > > > Fifth try:
> > > > 3518 ms
> > > > Query
> > > > urn:ogf:network:node=192.16.166.26
> > > > Services
> > > >
> > > > And on subsequent attempts no services were retrieved as well
> > > >
> > > > Obviously, we need to think of a systematic testing approach.