perfsonar development work

[Yee-Ting Li <>]

Hey Roman,

i tried installing the latest version (most of the problems i had  
were the mysql permissions)... i can log onto exist, i ensured that i  
do not have any other MA service or LS service running on the  
tomcat... everything goes well... until i initiate the exist db:

=====
[ytl@net-desk1:/opt/perfSONAR/current-SQL_MA/perfsonar/ant]$ ant -f  
build-sqlma.xml xmldb-init-sqlma
Buildfile: build-sqlma.xml

xmldb-init-sqlma:

getresource-rrdma:

xmldb-exist-rrdma:

BUILD FAILED
/afs/slac.stanford.edu/g/scs/net/netmon/perfSONAR/SQLtypeMAService/ 
java/20060828/perfSONAR-SQL_MA-src-snapshot-20060828/perfsonar/ant/ 
sqlma/sqlma-exist-xmldb-targets.xml:29: The following error occurred  
while executing this line:
/afs/slac.stanford.edu/g/scs/net/netmon/perfSONAR/SQLtypeMAService/ 
java/20060828/perfSONAR-SQL_MA-src-snapshot-20060828/perfsonar/ant/ 
rrdma/rrdma-exist-xmldb-targets.xml:65: Class  
org.apache.tools.ant.taskdefs.ConditionTask doesn't support the  
nested "exist" element.

Total time: 1 second
====

i have a log of my actions at:

https://confluence.slac.stanford.edu/x/Q0Q

any suggestions?

Yee.



On 30 Aug 2006, at 00:39, Roman Lapacz wrote:

> Yee-Ting Li wrote:
> > Hey Roman,
>
> H Yee,
>
>
> could you use the newest snapshot of SQL MA (perfSONAR-SQL_MA-src- 
> snapshot-20060828.tar.gz) and run the installation from the  
> beginning (some bugs were removed, also in the configuration script).
>
>
> Thanks for your work. I'm looking forward seeing your comments and  
> service running OK.
>
>
> Roman
>
>
> > Sorry for the late response - i've finally managed to get a little  
> > time to have a lot at this again.
> >
> > i've tried following your SQL MA install instructions as per the  
> > wiki. however, when it comes to the xmldb-init-sqlma ant target, i  
> > get:
> >
> > [ytl@net-desk1:/opt/perfSONAR/perfSONAR-SQL_MA-src- 
> > snapshot-20060811/perfsonar/ant]$ ant -f build-sqlma.xml xmldb- 
> > init-sqlma
> >
> > xmldb-init-sqlma:
> >
> > getresource-rrdma:
> >
> > xmldb-exist-rrdma:
> >     [exist] Checking collection: xmldb:exist://net- 
> > desk1.slac.stanford.edu:8680/exist/xmlrpc/db/sqlmaconfig
> >
> > __xmldb-init-rrdma:
> >
> > xmldb-init-rrdma:
> >
> > getdbroot-rrdma:
> >
> > get-xmldb-collection:
> >
> > xmldb-collection-rrdma:
> >    [create] Database driver already registered.
> >
> > BUILD FAILED
> > /afs/slac.stanford.edu/g/scs/net/netmon/perfSONAR/SQLtypeMAService/ 
> > java/20060811/perfSONAR-SQL_MA-src-snapshot-20060811/perfsonar/ant/ 
> > sqlma/sqlma-exist-xmldb-targets.xml:29: \
> >   The following error occurred while executing this line: \
> > /afs/slac.stanford.edu/g/scs/net/netmon/perfSONAR/SQLtypeMAService/ 
> > java/20060811/perfSONAR-SQL_MA-src-snapshot-20060811/perfsonar/ant/ 
> > rrdma/rrdma-exist-xmldb-targets.xml:147: \
> >   The following error occurred while executing this line: \
> > java.lang.NoSuchMethodError: org.exist.xmldb.XmldbURI.xmldbUriFor 
> > (Ljava/lang/String;)Lorg/exist/xmldb/XmldbURI;
> >
> > the exist.jar is in the classpath; but i still can't get the thing  
> > to work! help!
> >
> > i've documented my actions at:
> >
> > https://confluence.slac.stanford.edu/display/IEPM/IEPM-BW+MA
> >
> >
> > any insight is much appreciated! thanks,
> >
> > Yee.
> >
> >
> >
> > On 24 Aug 2006, at 04:08, Roman Lapacz wrote:
> >
> > > Yee-Ting Li wrote:
> > > > Hi,
> > >
> > > Hi Yee
> > >
> > >
> > > To add new measurement data to SQL MA you need to know this:
> > >
> > > - There must be a metadata configuration file which contains  
> > > pairs of metadata and data elements. Metadata elements describe  
> > > measurements (in case of the utilization the interface - 
> > > measurement source - is described; see example conf/sql- 
> > > database_utilization_TEST.xml). Data elements contain keys which  
> > > locate measurement results.
> > >
> > > Example snippet :
> > >
> > >    <nmwg:metadata id="meta1-test">
> > >        <netutil:subject id="subj1">
> > >            <nmwgt:interface>
> > >                <nmwgt:hostName>test-hostName</nmwgt:hostName>
> > >                <nmwgt:ifAddress type="ipv4">10.1.2.3</ 
> > > nmwgt:ifAddress>
> > >                <nmwgt:ifName>test-0</nmwgt:ifName>
> > >                <nmwgt:ifDescription>test description</ 
> > > nmwgt:ifDescription>
> > >                <nmwgt:direction>in</nmwgt:direction>
> > >                <nmwgt:authRealm>TestRealm</nmwgt:authRealm>
> > >                <nmwgt:capacity>1000BaseT</nmwgt:capacity>
> > >            </nmwgt:interface>
> > >        </netutil:subject>
> > >      <nmwg:parameters>
> > >        <nmwg:parameter name="supportedEventType">utilization</ 
> > > nmwg:parameter>
> > >      </nmwg:parameters>
> > >    </nmwg:metadata>
> > >
> > >
> > >    <nmwg:data id="data1-test" metadataIdRef="meta1-test">
> > >      <nmwg:key>
> > >        <nmwg:parameters>
> > >          <nmwg:parameter name="metadataId">meta1-test</ 
> > > nmwg:parameter>
> > >          <nmwg:parameter name="ibatisConfig">ibatis-SqlMapConfig- 
> > > utilization.xml</nmwg:parameter>
> > >        </nmwg:parameters>
> > >      </nmwg:key>
> > >    </nmwg:data>
> > >
> > > (any parameters in the key are provided to ibatis engine and can  
> > > be used in sql statements; in the example above metadataId points  
> > > measurement results in database table for the relevant metadata)
> > >
> > >
> > > - There must be database tables with measurement results. JDBC  
> > > parameters are located in ibatis files (see data elements in the  
> > > metadata configuration file). These files contain references (see  
> > > sqlMap element) to the next ibatis files with sql statements  
> > > (they will be used for fetching data).
> > >
> > > Example snippet:
> > >
> > >    <transactionManager type="JDBC">
> > >        <dataSource type="SIMPLE" >
> > >            <property name="JDBC.Driver"  
> > > value="com.mysql.jdbc.Driver"/>
> > >            <property name="JDBC.ConnectionURL"  
> > > value="jdbc:mysql://10.20.30.40/perfsonar_ma"/>
> > >            <property name="JDBC.Username" value="perfsonar_ma"/>
> > >            <property name="JDBC.Password" value="xyz"/>
> > >        </dataSource>
> > >    </transactionManager>
> > >
> > >    <sqlMap resource="ibatis-utilization.xml"/>
> > >
> > >
> > >
> > > - No need to create a new storage manager for SQL MA.
> > >
> > > - SQL MA needs to know how to fetch metadata/data from the  
> > > metadata configuration file. For this, a separate class which  
> > > generates xquery statement is needed. Take a look at the example  
> > > for the utilization:
> > >
> > > org.perfsonar.service.measurementArchive.metadataConfig.queryGenerat 
> > > or.UtilizationQueryGenerator
> > >
> > > - The service must send the response to the client application  
> > > and it is generated in a separate class. Example for the  
> > > utilization:
> > >
> > > org.perfsonar.service.measurementArchive.sqlType.responseGenerator.U 
> > > tilizationResponseGenerator
> > >
> > > - Above two classes must be related with eventType (you can find  
> > > eventType in the metadata configuration file and requests sent to  
> > > the services; in case of the utilization metric the value of  
> > > eventType is 'utilization').  The configuration of it is located  
> > > in conf/eventType-map.xml.
> > >
> > > Example snippet:
> > >
> > > <eventType>
> > >  <name>utilization</name>
> > >  <metadataQueryGenerator>
> > >     
> > > org.perfsonar.service.measurementArchive.metadataConfig.queryGenerat 
> > > or.UtilizationQueryGenerator
> > >  </metadataQueryGenerator>
> > >  <responseGenerator>
> > >     
> > > org.perfsonar.service.measurementArchive.sqlType.responseGenerator.U 
> > > tilizationResponseGenerator
> > >  </responseGenerator>
> > > </eventType>
> > >
> > >
> > >
> > >
> > >
> > > For the beginning the best way is the installation of SQL MA with  
> > > the utilization and L2 path status data.  Follow the installation  
> > > guide:
> > >
> > > http://monstera.man.poznan.pl/jra1-wiki/index.php/ 
> > > Java_SQL_MA_Service_Installation_Guide
> > >
> > > This will show you how it works and help you to understand better  
> > > what I've described above. If you have other questions, please,  
> > > don't hesitate to send them.
> > >
> > >
> > > Roman
> > >
> > >
> > > > in order to better understand the architecture of perfSONAR, we  
> > > > have decided to create a MA for our IEPM-BW data. iepm-bw  
> > > > basically an end-to-end data monitoring suite. we have data  
> > > > stored all over the world in separate MySQL databases from which  
> > > > we would like to plug into perfSONAR.
> > > >
> > > > We have separate tables for almost every kind of measurement;  
> > > > pings, traceroutes, iperfs, pathchirps etc etc. in order to  
> > > > better understand how i should implement the interface, i would  
> > > > appreciate it if you could answer some questions for me.
> > > >
> > > > - i understand that there is a SQLStorageManager that already  
> > > > exists and uses the ibatis framework for easy SQL data  
> > > > retrieval. would it make more sense to write xml ibatis config  
> > > > files to query for our mysql data rather than implement our own  
> > > > MA in its entirety?
> > > >
> > > > - correct me if i'm wrong; one uses a key to query for the  
> > > > relevant data, ie the examples have a ibatis config pointed to  
> > > > it in the request for data (which ibatis links to a table with  
> > > > bw data). if i were to implement a single MA for the IEPM-BW  
> > > > data, i'm guessing the client will need to discover which ibatis  
> > > > config would lead to each type of network metric  
> > > > (characteristic). clients can also query for keys, and or just  
> > > > ask for certain types of information (which the storagemanager  
> > > > would map to the relevant data source)
> > > >
> > > > - how does the mapping for certain metrics work? how closely  
> > > > tied in with nmwg is this? ie if i request bandwidth  
> > > > information, are there any systems/code in place to basically  
> > > > map to say iperf information? is there a registration process  
> > > > for the types of metrics which an MA provides?
> > > >
> > > > - there is a 'generic perl sql' implementation mentioned in the  
> > > > wiki; which basically describes how typically all that is  
> > > > required is a datestamp and value for a characteristic. how does  
> > > > the type of metric fit into this, and how is this supplied in  
> > > > the response? (kinda asking the same question as the previous  
> > > > question).
> > > >
> > > > - is there a dynamic or static method of invocation of the  
> > > > StorageManager implementations? how do i add new StorageManagers  
> > > > to perfSONAR? how do i check to see if perfSONAR has a specific  
> > > > type of StorageManager invoked?
> > > >
> > > > Thanks!
> > > >
> > > > Yee.
> > >
> > >
> > > --
> > > //  PSNC, Poland
> > > //  phone: (+48 61) 858 20 24
> > > //  http://www.man.poznan.pl
>
>
> --
>
> //  PSNC, Poland
> //  phone: (+48 61) 858 20 24
> //  http://www.man.poznan.pl