perfsonar development work

["Vedrin Jeliazkov" <>]

Hi again,

Some further testing revealed that the incorrect MetadaKeyResponse problem is
not exhibited when SVN RRD MA is configured to communicate with eXist via
xmlrpc. I was preparing to run some tests against older eXist versions, when I
discovered that more strings get stripped if SVN RRD MA is not configured to
use eXist at all. Here is a brief summary:

- SVN RRD MA + eXist + xmlrpc -> OK;
- SVN RRD MA + eXist + http -> 2 errors (ifName and ifDescription);
- SVN RRD MA without eXist -> 4 errors in ifAddress;

All tests share a common RRD config file and the same MetadataKeyResponse
errors reappear in subsequent runs.

Kind regards,
Vedrin

PS: We'll keep using eXist via xmlrpc until a fix becomes available.

"Vedrin Jeliazkov" 
<>
 wrote:

> Hi again,
> 
> After upgrading eXist from 1.1rc-20060706 to 1.1rc2-20060901 the
> SetupDataRequest (without key) processing performance is back to normal. It
> looks like there was some performance hog, exhibited only when running
recent
> SVN RRD MA together with eXist 1.1rc-20060706. So this issue should be
> considered resolved.
> 
> However, the other problem, related to stripped strings in
MetadataKeyResponse
> (explained in more detail in my previous mail) remains open.
> 
> Kind regards,
> Vedrin
> 
> "Vedrin Jeliazkov" 
> <>
>  wrote:
> 
> > Hi Roman,
> > 
> > I'm coming back to an issue we discussed about 2 weeks ago. We've tried
> > replacing
> > 
> > <nmwg:eventType>utilization</nmwg:eventType>
> > 
> > with
> > 
> > <nmwg:parameters>
> >   <nmwg:parameter name="supportedEventType">utilization</nmwg:parameter>
> > </nmwg:parameters>
> > 
> > in MetadataKeyRequests and SetupDataRequests sent by perfsonarUI to 
> > the SVN (2006-09-06 22:05) version of MA. In both cases the service
returns
> > the following error:
> > 
> > MetadataQueryGeneratorFactory.getMetadataQueryGenerator: eventType  is not
> > supported
> > 
> > This might be normal behaviour and in fact we are happy that we probably
> don't
> > have to change the request format (could you confirm this?).
> > 
> > However, this also means that the solution for the significant slowdown of
> > SetupDataRequest processing should be searched elsewhere. Do you have any
> > ideas?
> > 
> > The same holds true for the second problem, related to the incorrect
> > MetadataKeyResponse returned by the SVN version of MA. In our specific
> example
> > the SVN MA service unexpectedly returns two stripped strings in the
> > MetaDataKeyResponse:
> > 
> > 1) <nmwgt:ifName>.6</nmwgt:ifName> instead of
> > <nmwgt:ifName>Fa0/0.6</nmwgt:ifName> in <nmwg:metadata id="meta73">;
> > 
> > 2) <nmwgt:ifDescription>ORE-SOF(3)</nmwgt:ifDescription> instead of
> > <nmwgt:ifDescription>SW-gecko==CORE-SOF(3)</nmwgt:ifDescription> in
> > <nmwg:metadata id="meta188">;
> > 
> > As perfsonarUI relies on matching ifName and ifDescription (among others)
> for
> > grouping in/out data for the same interface, the incorrect
> MetadataKeyResponse
> > leads to an erroneous list of 80 (instead of 78) interfaces displayed.
> > 
> > Removing the 2 interfaces (both directions) from the config file doesn't
> solve
> > the problem - in such case some other (previously correct) strings get
> > stripped.
> > 
> > I'm attaching:
> > 
> > - the RRD database config file of perfSONAR-1.0 MA;
> > - the RRD database config file of SVN MA;
> > - the MetadataKeyRequest (identical in the two cases);
> > - the MetadataKeyResponse of perfSONAR-1.0 MA;
> > - the MetadataKeyResponse of SVN MA (some unexpectedly stripped strings,
> > mentioned above);
> > 
> > Should I create entries in bugzilla for these 2 issues (SetupDataRequest
> > processing slowdown and incorrect MetadataKeyResponse in SVN (2006-09-06
> > 22:05) version of MA)?
> > 
> > Best regards,
> > Vedrin
> > 
> > PS: The Python version of the service also returns the expected correct
> > MetadataKeyResponse when configured with
rrd-database_ISTF-perfSONAR-1.0.xml
> > 
> > "Vedrin Jeliazkov" 
> > <>
> >  wrote:
> > 
> > > Hi Roman,
> > > 
> > > Roman Lapacz 
> > > <>
> > >  wrote:
> > > 
> > > > Vedrin Jeliazkov wrote:
> > > > > [To the particular attention of CARNET, CESNET, FERMILAB, GEANT,
> > > Internet2,
> > > > > MREN, PIONIER, RNP and UNINETT]
> > > > >
> > > > > perfSONAR interface utilization MA status report 20060907(*)
> > > > >
> > > > > (*) The tests were run through a high capacity (100 Mb/s) low
latency
> > link
> > > > > with perfsonarUI-v0.08
> > > > >
> > > > > Notes:
> > > > >       1) 8 services out of a total of 17 have various problems 
> > > > > listed
below
> > > > >       (availability of only ~50% of the services is not good news);
> > > > >
> > > > >       2) at least 4 of the services are still running old versions
> > > > >       (the deadline for upgrades was 17 Aug 2006);
> > > > >
> > > > >       3) SetupDataRequest (without key) processing is up to 10 times
longer
> > > > >       in SVN, compared to perfSONAR-1.0;
> > > > >   
> > > > Vedrin,
> > > > 
> > > > I understand you are testing both versions on the same PC
> > > 
> > > Yes - same PC and same settings as well. The only difference I can think
> of
> > is
> > > that the SVN version of RRD MA is configured to communicate with eXist
via
> > > http, while perfSONAR-1.0 uses xmlrpc. Reverting to xmlrpc in the SVN
> > version
> > > doesn't help (this is normal, but I wanted to make sure that this is not
> the
> > > real issue).
> > > 
> > > One strange thing I've noticed is that perfsonarUI consistently reports
78
> > > interfaces when querying both Python version and perfSONAR-1.0 version,
> but
> > > erroneously reports 80 interfaces when querying the SVN version. We
> haven't
> > > traced this bug yet - it might be in client's code. However, it's a bit
> > > strange that it is reproducible only with the SVN version. The only
> > difference
> > > in the config files used by the respective services is related to the
> > required
> > > supportedEventType syntax change. While speaking about this, I realised
> that
> > > perfsonarUI also sends an old fashioned eventType syntax in its
requests.
> > > We’ll have to check if updating the request would resolve the
> > SetupDataRequest
> > > performance issue as well – I’ll let you know when we obtain some
results.
> > > 
> > > Kind regards,
> > > Vedrin
> > > 
> > > > 
> > > > Roman
> > > > 
> > > > 
> > > > >
> > > > > Legend:
> > > > >       (1) Number of Interfaces (NoI)
> > > > >       (2) (Full MetadataKeyRequest)/NoI [msec] 
> > > > >       (3) Average SetupDataRequest (without key) [msec]
> > > > >       (4) Average Link latency (RTT) [msec]
> > > > >       (5) Version
> > > > >
> > > > >               (1)     (2)     (3)     (4)     (5)
> > > > >
> > > > > CARNET                -       -       -       57      Python 0.1-15
> > > > > CESNET                1       2500    4000    -       perfSONAR-1.0 
> > > > > (restricted)
> > > > > ESNET         419     131     990     225     perfSONAR-1.0
> > > > > FERMILAB      -       -       -       176     Java?
> > > > > GEANT         42      71      429     85      Java 0.1 (EGEE tag)
> > > > > GRNET         32      78      313     90      perfSONAR-1.0
> > > > > Internet2     83      482     2850    195     Java 0.1 (EGEE tag)
> > > > > ISTF-J                78      51      3327    0.201   SVN from 
> > > > > 2006-09-06 22:05 EEST
> > > > > ISTF-P                78      38      487     0.201   Python 0.1-20
> > > > > MREN          2       351     1250    -       Java snapshot 20060214
> > > > > PIONIER-RRD   -       -       -       74      perfSONAR-1.0
> > > > > PIONIER-SQL   -       -       -       74      snapshot from 
> > > > > 2006-09-01
> > > > > RNP-J         33      121     818     313     perfSONAR-1.0
> > > > > RNP-P         63      63      889     313     Python 0.1-19
> > > > > SEEREN                7       85      189     14      perfSONAR-1.0
> > > > > SWITCH                330     64      380     74      perfSONAR-1.0
> > > > > UNINETT               -       -       -       95      Python 0.1-19
> > > > >
> > > > > Services which should be fixed:
> > > > >
> > > > > CARNET                java.net.ConnectException: Connection timed 
> > > > > out: connect
> > > > >
> > > > > CESNET                
> > > > > org.perfsonar.service.commons.exceptions.SystemException:
> > > > >               [unknown]: RRDStorageManager.rrdFetch:
> > > > >               Cannot connect with rrdjtool native library:
> > > > >               no jrrd in java.library.path
> > > > >
> > > > > FERMILAB      Returns an empty result set
> > > > >               Service description missing on the wiki
> > > > >
> > > > > Internet2     
> > > > > org.perfsonar.service.commons.exceptions.SystemException:
> > > > >               Cannot retrieve data from rrdjtool library:
> > > > >               RRDFETCH failed:
> > > > >               opening '/home/sonar/rrd-data/hstn-northtexas.rrd':
> > > > >               No such file or directory
> > > > >
> > > > >               RRDs should be updated more frequently
> > > > >
> > > > > PIONIER-RRD   java.lang.NullPointerException
> > > > >
> > > > > PIONIER-SQL   MetadataConfigurationStorageManager.fetch: Connection 
> > > > > to
> xml
> > > > > database failed
> > > > >
> > > > > RNP-P         RRDs not updated with utilization data since 30 Jul 
> > > > > 2006
> > > > >
> > > > > UNINETT               java.net.ConnectException: Connection 
> > > > > refused: connect
> > > > >
> > > > >   
> > > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> 
>