perfsonar development work

[Loukik Kudarimoti <>]

Hi,

I spent some time today explaining pS concepts and profiles to Ivan. We 
had an IM chat and I thought that we could make use of this chat log for 
similar introductions to other new developers.

The pS profile that I tried explaining is that of RRD MA. I explained 
how the service handles setupdatarequest and metadatakey request and 
which components get invovled, which components are ps-base and which 
components to be implemented and so on. (I think we should start naming 
our profiles)

Comments and re-use of logs most welcome :)

Loukik.
------------------------------------------------
Start of Ivan Koga buffer: Wed Aug 23 15:02:47 2006
[13:53] Ivan Koga: Hi Loukik
[13:53] loukik: hello ivan
[13:53] Ivan Koga: :) How are you?
[13:53] loukik: I am alright. what about you?
[13:53] Ivan Koga: fine :)
[13:53] Ivan Koga: Do you know what I have to do next with AMP-MA?
[13:54] Ivan Koga: the example of messages are quite ok... 
[13:54] Ivan Koga: but i think the next will be define the relaxNG shemas... 
but i dont know 
[13:54] loukik: ok, if there are new elements in your messages, you have to 
make sure jason has added them to the message object structure
[13:55] loukik: and after that, its time to start building your service
[13:55] Ivan Koga: I think the messages are ok
[13:56] Ivan Koga: Jason is on vacation 
[13:56] Ivan Koga: I am just asking because i wanted to do something... to 
not stop this work
[13:56] loukik: i agree..
[13:56] loukik: have you had a look at the RRD ma or the SQL MA?
[13:57] loukik: basically, you need to start designing the blocks..
[13:58] loukik: this if ofcourse based on the functionality that you want to 
offer
[13:58] loukik: if you think the 'profile' of RRD MA suits the profile that 
you want for AMP MA, 
[13:59] loukik: you can design your blocks similar to that of RRD MA
[13:59] Ivan Koga: Blocks = Datum, parameters, subject?
[13:59] loukik: no no..
[13:59] loukik: blocks as in components
[13:59] loukik: components of the service
[14:00] loukik: have you had a look at the RRD MA's components?
[14:02] Ivan Koga: no
[14:02] loukik: I can explain briefly explain it to you now..but i suggest 
you have a look as i believe it will be similar
[14:03] loukik: RRD MA handles two types of requests
[14:03] loukik: MetadataKeyRequest and SetupDataRequest
[14:04] loukik: A metadataKeyRequest helps in finding out the metadata of 
interest (based on matching the request message against an xml config file)
[14:04] loukik: for each successive match of metadata, it gives back a key. 
The key contains the location information for the data
[14:04] loukik: for example
[14:05] loukik: the request might say that it is look for utilisation data 
for an interface with IP address 193.63.211.130
[14:06] loukik: the request is matched against the config file to see if 
there are any metadata elements in the config which contains the same 
interface IP address
[14:06] loukik: (are you still with me?)
[14:07] Ivan Koga: yes
[14:07] Ivan Koga: i am trying to understand :P
[14:07] loukik: if you have any questions, ask them immediately. Ok?
[14:08] Ivan Koga: ok
[14:08] loukik: so, the rrd ma looks for the match in its config file to see 
if the interface ip address is present..
[14:08] Ivan Koga: if there is no matched IP, it returns null?
[14:09] loukik: if there are no matches, the service returns an error code 
saying "no matching metadata found"
[14:09] Ivan Koga: ok
[14:10] loukik: the config file contains metadata information such as 
interface ip address, host name, interface name, etc
[14:10] loukik: so, the user's request can contain any of these.
[14:10] loukik: each metadata (or a chain of metadata) in the config file is 
linked to a data element in the config file..
[14:10] loukik: this data element contains a key
[14:11] loukik: for example, the key in rrd ma is a filename and datasource 
pair
[14:11] loukik: from the user point of view, this key is useless
[14:11] Ivan Koga: ok 
[14:11] loukik: or he/she doesn't know whats in the key..its opaque
[14:12] loukik: from the service point of view, this key is absolutely 
essential because it tells the service which file to look into
[14:12] loukik: without the key, the service won't work (at least the rrd ma)
[14:13] Ivan Koga: So, If I want data from a period of time, from all the 
interfaces in the RRD-MA, I would pass the start-time and end-time, and it 
will return all the keys that match this?
[14:13] loukik: that is more or less equivalent to an empty metadata request
[14:14] loukik: the config file doesn't store the start time and the end time
[14:14] loukik: (because its too dynamic)
[14:14] loukik: well, at least as of now, it doesn't store the start time and 
end time. don't know about the future
[14:14] loukik: so, your question is...
[14:14] loukik: will an empty metadata give me back all the keys
[14:14] loukik: the answer is..
[14:15] loukik: it depends on how the service is configured
[14:15] Ivan Koga: ok
[14:15] loukik: i can't remember about rrd ma
[14:15] loukik: but for nms type mp (which uses similar profile)
[14:15] loukik: it works...i.e. it gives back all the keys
[14:15] loukik: so, its upto the desinger of the service to put in that 
feature..
[14:15] Ivan Koga: I think, in MA-AMP, it will return nothing
[14:16] loukik: as the designer should consider the positive and negative 
aspects
[14:16] loukik: ok, giving back everything could be very powerful in some 
cases
[14:16] loukik: but a bad decision for others
[14:16] loukik: its upto you
[14:16] Ivan Koga: do you know where is the code that does this interaction? 
[14:16] loukik: aha....
[14:16] loukik: you asked the right question ;)
[14:17] Ivan Koga: the XML interation between the user and the service]
[14:17] loukik: umm..there is more to this..hold on..
[14:17] Ivan Koga: ah, ok! :)
[14:17] loukik: we will come to that soon
[14:17] loukik: so, a metadata key request tells the service..
[14:18] loukik: "do the match based on the elements in the request, give back 
all other elements present in the matching metadata elements and also give 
back linked data elements"
[14:19] loukik: so, we call it Metadata and key request (well, we removed the 
and to make it simpler - MetadataKeyRequest)
[14:19] loukik: so, the service only does the match and returns the match 
along with the keys..
[14:19] loukik: it doesn't return any data
[14:19] loukik: is this clear?
[14:20] Ivan Koga: yes
[14:21] loukik: ok, now the SetupDataRequest
[14:21] loukik: the setupDataRequest consists of two sub-types
[14:21] loukik: sub-type 1 does not contain any key but only metadata
[14:21] loukik: sub-type 2 contains a key
[14:21] loukik: (you have to note that a key is returned in the data element 
in a MetadataKeyResponse_
[14:21] loukik: )
[14:22] loukik: but in SetupDataRequest, the key is given in the metadata 
element (data element is empty)
[14:23] loukik: so, for sub-type 1 (no key), the service first does the exact 
procedure as it would do for MetadatakeyRequest...
[14:23] loukik: i.e., it tries to find a match 
[14:23] Ivan Koga: humm
[14:23] loukik: after it gets the match and gets the key..
[14:23] loukik: instead of returning the match and key to the user
[14:24] loukik: it uses the key (or keys depending on the number of matches) 
to fetch data 
[14:24] loukik: and returns the data to the user
[14:24] Ivan Koga: ok, i understand
[14:24] Ivan Koga: so, the sub-type 1 does not need the metadatakeyrequest
[14:24] loukik: ?
[14:25] loukik: sub-type 1 follows 50% of metadatakey request
[14:25] Ivan Koga: the sub-type 2 is supposing that it does the 
metadatakeyrequest at first
[14:25] loukik: but instead of passing the response (i.e the other 50% ), it 
passes the data back
[14:25] loukik: sub-type 2 assumes that the user/client has already done the 
metadatakey request
[14:26] loukik: sub-type 1 assumes that the user/client hasn't done the 
metadatakey request before..so, it does it first before fetching the data
[14:27] loukik: so, for the service, finding out the key is very important..
[14:27] loukik: the question is..whether the user does it 
[14:27] Ivan Koga: so, let me see if I understood
[14:27] Ivan Koga: to use the sub-type2, I have to make a metadatakeyrequest 
and then setupdatarequest to have the data
[14:28] Ivan Koga: to use the sub-type1, I have only to make a 
setupdatarequest
[14:28] loukik: yup, thats right
[14:28] Ivan Koga: but what is the advantage to have this separation?
[14:28] loukik: sub-type 1 takes care of people not interested in using a key
[14:29] loukik: sub-type 2 provides efficiency because no metadata match is 
required
[14:29] loukik: since the key is already there, the service simply returns 
data for that key
[14:30] loukik: we believe that users who rarely use the service will use 
sub-type 1
[14:30] loukik: and clients such as weathermap will use sub-type 2
[14:30] loukik: because weathermap needs efficiency
[14:30] Ivan Koga: oh! I think I understand now
[14:32] Ivan Koga: If a user already know that he wants data from an IP, he 
should make a key request, store it and then setupdatarequest sub-type 2 and 
it will only make the metadatakeyrequest once... and He will get the data 
faster
[14:32] loukik: you got it!
[14:32] loukik: well, setupdata request sub-type 2 will never try to find a 
key because its already there!
[14:32] loukik: the user will use metadata key request and find a key and 
store it
[14:33] loukik: if the key is passed in a setupdatarequest
[14:33] loukik: then it becomes sub-type2
[14:33] loukik: if no key is passed, it becomes sub-type 1(i.e go and do a 
match and get a key)
[14:33] loukik: clear?
[14:34] Ivan Koga: yes
[14:34] loukik: so, now the components
[14:34] loukik: all requests come to RequestHandler
[14:34] loukik: they come in org.w3c.dom.Document format
[14:34] loukik: RequestHandler is a commmon component
[14:35] loukik: ie similar across all services
[14:35] Ivan Koga: ok
[14:35] loukik: so, you will need to re-use it
[14:35] Ivan Koga: so, i dont need to change it 
[14:35] loukik: The RequestHandler calls on MessageHandlerFactory to find out 
a message for the given request
[14:35] loukik: no, you don't need to make changes to the request handler
[14:36] loukik: sorry, let me repeat
[14:36] loukik: th requesthandler calls on messageHandlerfactory to get hold 
of a messagehandler for the request
[14:37] loukik: the messageHandlerfactory makes use of config files (to see 
which messages are supported) and provides a suitable message handler
[14:37] Ivan Koga: ah, ok
[14:37] loukik: if your are using standard message types (metadatakey request 
and setupdata request)
[14:37] loukik: then you don't need to worry about message handler either
[14:37] loukik: if your message types are complicated (like LS)
[14:38] loukik: you will need a new message handler.
[14:38] loukik: so, for now, lets assume that you don't need one..
[14:38] Ivan Koga: ok
[14:38] loukik: we can investigate this later
[14:38] loukik: so, each message handler looks into the configuration to find 
out which service engine implementation to use
[14:38] loukik: so, basically, message handlers call ont he service engine
[14:39] loukik: a serviceengine is a java interface
[14:39] loukik: so, you need to implement it
[14:39] loukik: for your service
[14:39] loukik: a serviceengine implementation is the heart of any service
[14:39] loukik: which decides what to do to satisfy each request
[14:40] loukik: so, if the request is of get key
[14:40] loukik: i.e. metadatakeyrequest
[14:40] loukik: it uses a configuration storage manager to do a match against 
the request
[14:40] loukik: if the request is for data
[14:41] loukik: it checks if there is a key already
[14:41] loukik: if yes, it calls on a storage manager to fetch data from the 
rrd file
[14:41] Ivan Koga: is it a general component too? 
[14:41] Ivan Koga: generic?
[14:42] loukik: well, the component which can talk to an xml database is 
generic
[14:42] loukik: but this component requires an xpath or xquery expression
[14:42] loukik: so that it can talk to the xml database
[14:42] loukik: that needs to be constructed by the service engine
[14:43] Ivan Koga: so, i have to implement an storage manager to fetch my 
data from AMP files
[14:43] loukik: yes, you have to implement a storage manager for amp files
[14:43] loukik: we have RRDTypeStorageManager
[14:43] loukik: so, we need AMPTypeStorageManager
[14:44] loukik: now, this AMPTypeStorageManager only does reading of AMP files
[14:44] loukik: for the configuration (metadata matching)
[14:44] loukik: you need existDB storage manager
[14:44] loukik: which is already present
[14:44] loukik: so, just re-use it
[14:45] loukik: but what you need to do is 
[14:45] loukik: construct xquery expression
[14:45] loukik: because existDB storage manager requires xquery expression
[14:45] loukik: its not soo difficult :)
[14:46] Ivan Koga: Will existDB will store the name of the files?
[14:46] Ivan Koga: the "keys"
[14:47] loukik: umm...
[14:48] loukik: exist db assumes that a config file is already there
[14:48] loukik: i.e., you write up a config file and store in the exist db 
during service installation
[14:48] loukik: exist db then does xquery based matching against this file
[14:49] loukik: check your email..
[14:49] loukik: i just sent 3 emails with uml diagrams
[14:49] loukik: these are for snmp mp
[14:50] loukik: just replace the snmp measurement executor with your amp 
storage manager
[14:50] loukik: and it could be the profile for your service
[14:50] Ivan Koga: ok
[14:50] loukik: snmp mp and rrd ma are very very similar
[14:51] loukik: oh, i forgot to mention
[14:51] loukik: request handler gets a org.w3c.dom.Document object
[14:51] loukik: using xmlutils, it converts it into Message object
[14:51] loukik: Message object is our data container 
[14:51] loukik: its consists of one class per element name
[14:52] loukik: and provides getter and setters
[14:52] loukik: we use message object throughout our service
[14:53] loukik: i suggest that you look at rrd ma sources or snmp mp sources
[14:53] loukik: and see if you can get the next picture
[14:53] Ivan Koga: What is stored in the existDB? The keys? What kind of 
information? 
[14:53] loukik: umm..i can point you to an example
[14:54] loukik: 
http://monstera.man.poznan.pl/jra1-wiki/index.php/RRD_description_file
[14:54] Ivan Koga: <nmwg:Data id="data1" metadataIdRef="meta1">

   <nmwg:key id="localhost.-77179f19:10b5b954cea:-7e4f">

     <nmwg:Parameters id="localhost.-77179f19:10b5b954cea:-7e4e">

       <nmwg:Parameter name="dataSource">ds0</nmwg:Parameter>

       <nmwg:Parameter

name="file">/PATH/TO/SOME_RRD_FILE.rrd</nmwg:Paramete r>

     </nmwg:Parameters>

   </nmwg:key>

 </nmwg:Data>
[14:54] loukik: see the big picture at the end
[14:55] Ivan Koga: something like this?
[14:55] loukik: yes
[14:55] loukik: but linked to a metadata
[14:56] loukik: ok, i have to get back to other stuff now..
[14:56] loukik: i will email this chat conversation to ps-dev
[14:56] loukik: so that we can use it for some other new developer ;)
[14:56] loukik: notice <nmwg:store 
[14:56] loukik: in the big picture xml
[14:57] loukik: for messages this is usually 
[14:57] loukik: <nmwg:message
[14:57] loukik: for store it is <nmwg:store
[14:57] loukik: store=config
[14:58] Ivan Koga: hum... ok
[14:59] loukik: any more questions?
[15:00] Ivan Koga: so, i need to take a loot at the RRDTypeStorageManager 
[15:00] loukik: yup and also the complete service 
[15:01] Ivan Koga: i think i can make some associations around the RRD-MA to 
fit AMP-MA
[15:01] Ivan Koga: if i have any question can i ask you?
[15:01] loukik: sure...
[15:02] loukik: if you are emailing, put a cc to perfsonar-dev
End of Ivan Koga buffer: Wed Aug 23 15:02:47 2006