perfsonar development work

[Loukik Kudarimoti <>]

Hi Jeff, Ivan,

I was responsible for a new developer document but I never managed to 
dedicate enough time to complete it. We managed to put some things on 
the wiki in the generic service section but that is pretty much where it 
stands :(

I am attaching a document which details the design of SNMP MP with the 
help of sequence diagrams (Refer to section 2.1.5 for SNMP MP). We wrote 
this up for our deliverable. The sequence diagrams illustrate how 
MetadataKeyRequest and SetupDataRequest messages are handled by the 
service. Its very similar to how RRD MA, SQL MA and NMS MP services work 
(I haven't looked into the details of other services).

My understanding is that there are a few "profiles" which can be used in 
order to develop a new service. The choice of profile is dependent on 
the functionality that is required (and possible) from the new service. 
I believe you have accurately explained one such profile which is used 
by CLMP. The attached document explains a different profile which is 
used by a few other services. I am not aware of the profile being used 
by the LS, ongoing work on Topology service and a few other Measurement 
Point services.

I believe all profiles will have an identical step 1 (including 1a, 1b 
and 1c) that you have mentioned below (these steps are not in my 
document). What is also common across most services is the creation of 
Message object from the incoming XML DOM object. This is taken care of 
by the RequestHandler component.  The details of Step 2 and later steps 
depends on each profile. For example, in SNMP MP, RRD MA and SQL MA, we 
make use of eXistXmlDBStorageManager (an implementation of storage 
manager interface) which lets the services interact with an XML database 
in order to satisfy MetadataKey Request Messages. The 
eXistXmlDBStorageManager requires a XPath or XQuery expression. Hence, 
each service has a dedicated class which is used to generate 
XQuery/XPath expressions depending on the contents passed to it.

When I started writing up the document, I started with a slightly 
different sequence than the one you have mentioned Jeff. The sequence was

1) Determine the functionalilties that you want to provide in your 
service (Basically choose a profile or come up with your own if none of 
them are suitable)

2) Propose a strawman for XML messages (give examples of how messages 
could look like) that will be exchanged between your service and the 
client. Martin and Jason are responsible for the approval of messages 
and creation of objects (if you need any new elements) within the beans 
that we use in our service in order to support these messages.

3) Each profile more or less illustrates the constituents of your new 
service and with the help of sequence diagrams, it also illustrates how 
these components will interact with one another. You should be able to 
figure out which classes you need to write up in order to complete your 
service and which classes (from perfSONAR base) you can re-use. You 
should also be able to figure out which java interfaces you need to 
implement.

4) An understanding of different configuration files 
(service.properties, components.properties, log4j.properties, 
const.properties) is required. It should be fairly simple to understand 
the role of each property file and what you need to do with them. During 
the design phase of your service, you should be able to figure out what 
properties go into each of these files and what default values you would 
associate with these properties.

5) Once you have determined which classes you need and which classes you 
can re-use, you can start implementing them.

6) You will then need to write up the JUnit classes for all the classes 
that you write.

7) You will then need write up a build-${X}.xml file and also need to 
create ant targets to
  a) build your service - includes creating jars
  b) make a tar ball for your service (binaries and sources)
  c) configure your service for installations
  d) deploy/undeploy your service
  c) test your service (some of us have ant targets to test major 
components within the service)
  You can do all the above by using some of the existing ant targets as 
an example ( I did a lot of copy-paste and it didn't take too much time)

These are the steps I could think of. I hope I haven't missed out on any.

Regards,
Loukik.


Jeff W. Boote wrote:

> Loukik,
>
> I have a vague memory of you saying you were going to document the 
> 'pieces' that needed to be implemented for someone to create a new 
> service. Is my memory correct? If so, what is the status. If not, do 
> you remember who said they would?
>
> Ivan,
>
> I will try and very roughly outline it here - but I am afraid I might 
> forget something. (Loukik, if you notice something missing or wrong, 
> please say so.)
>
> First off, I'm going to use the CLMP as a model because it is the 
> service I understand best at this point. There is some documentation 
> on it (including a sequence diagram that shows the classes that are 
> interacted with) at:
> http://wiki.perfsonar.net/jra1-wiki/index.php/CLMPService
>
> 1) Create a new build setup for your service:
> 1a) Create the ant directory for your service. Copy one of the 
> existing service directories including all the files (the rrdma is 
> probably the most 'tested' and functional of the build environments). 
> You will need to rename the targets in the file for your 'service' 
> name ${X}.
> 1b) Create the build-${X}.xml file
> 1c) Create the const.properties.${X} file.
> 2) Create the needed 'class files' for the phases of the 
> request/response:
>     Axis Dispatch Handler
>         You should have nothing to do here.
>     Message Handler
>         With any luck you can use the existing message handler class 
> for the MA messages. The two message types you should be using are 
> 'SetupDataRequest' and possibly 'MetadataKeyRequest'. If you can use 
> those, then you don't need to mess with the message handlers. (The 
> service.${X}.message_types property will determine what message 
> handlers are called.)
>     ServiceEngine
>         This you will likely have to create yourself. See the several 
> service engines in the source code for examples. 
> (perfsonar/src/org/perfsonar/service/measurementPoint/commandLine/ComamndLineMPServiceEngine.java 
> is just one.)
> The service.r.type property is used to find the correct service engine 
> class. It has to support specific 'action' types based on the message 
> types you accepted earlier.
>
> At this point your 'service engine' can communicate with AMP and 
> return the data. The other part you will need to deal with is of 
> course the schema.
>
> Please look at the CLMPService page I referenced above. Fausto wrote 
> some documentation indicating what needs to be added in the nmwg 
> classes to support that. As far as figuring out a good schema for the 
> data you are doing, please send email to the perfsonar-dev email list 
> and ask questions. There are several of us that have ideas for 
> specific types of data. It would probably be best to talk about this 
> in a larger group.
>
> jeff
> Ivan K. Koga wrote:
>
> > Hi Jeff,
> >
> > I continue to read the code and the document, but think I am not 
> > getting too much progress on it.
> > What i would like to know is if we want to develop a new MA, Do we 
> > have to develop all the classes like RequestHandler, MessageHandler, 
> > MessageHandlerFactory, ServiceEngine, ServiceEngineFactory... Or can 
> > we start with the RequestHandler(generic), MessageHandler(generic), 
> > and so on and modify only the classes that is more specific like 
> > ServiceEngine?
> > If it is true, could you point me the classes that are more specific 
> > and what is more generic? Do you have some kind of generic model to 
> > make my study easier? A runnable code, if it is possible.
> >
> > Thank you,
> > Ivan K. Koga
> >
> >
> >
> > Jeff W. Boote escreveu:
> >
> > > Ivan K. Koga wrote:
> > >
> > > > Hi Jeff,
> > > >
> > > > I´ve already implemented some classes that interact directly with 
> > > > AMP data and AMP configuration files. What I need now is the 
> > > > classes that make available the Web services with NMWG schema that 
> > > > will interact directly with my implemented classes.
> > > >
> > > > It´s a kind of MP and MA AMP but just to local hosts (Amplets). 
> > > > Maybe this could be useful to a implementation o MA (based on the 
> > > > AMP datacollector) if the data in the MA seems like the AMplet.
> > >
> > >
> > >
> > > Initially, as far as perfSONAR integration - I would probably start 
> > > with just making the AMP data available and not worry about the 
> > > configuration file part. (You have other ways of doing that for now.)
> > >
> > > > Do you know something about AMP?
> > > > The AMPlet have some configuration files that indicate where it 
> > > > will make the measurement, when it will occur, frequency, and so on.
> > > > After inserting the needed values into the configuration files, the 
> > > > AMPlet will look at this configuration files and start the 
> > > > measurements. After this it will write the measurement data.
> > >
> > >
> > >
> > > I know very little about the internals of AMP.
> > >
> > > > So, what i made is implement some classes that interact with this 
> > > > configuration files making available some services that schedule 
> > > > data. I also implemented a class that read the available 
> > > > measurement data and make it available.
> > >
> > >
> > >
> > > So, I would suggest a phased approach. First expose this part that 
> > > can read the available measurement data via the MA perfSONAR service 
> > > interfaces. Then, expose the other part. The MA is a little easier 
> > > and will be a good way to learn what is going on before trying to 
> > > expose the AMP configuration stuff.
> > >
> > > > I don´t know if  what i want is the same case as adding a new tool 
> > > > into the command line MP because AMP is not a command line tool. 
> > > > However, I will read fausto´s documentation and try to do something 
> > > > with it.
> > >
> > >
> > >
> > > I was not suggesting that you add AMP as a command-line tool. I was 
> > > suggesting that you look at the set of classes that were used to 
> > > create the command-line MP and that would help you understand what 
> > > would be needed to write a service for AMP...
> > >
> > > I still suggest that, but please ask me questions about it. The 
> > > problem here is that we don't have developers documentation - so the 
> > > best reference is other written services. I suggest the command-line 
> > > service because I know more about it than the other ones and it will 
> > > be easier for me to help.
> > >
> > > jeff

Attachment: DJ13.2 Phase II Implementation report v1.1-LK.doc
Description: application/ms-word