perfsonar development work

[Andreas Hanemann <>]

Dear all (especially Jeff, Martin, Loukik, Szymon and Nicolas),

For the exchange of data between the Hades MA and CNM we are currently 
facing the problem to cope with the amount of data that should be 
transferred. Even though we have tried to comply to the perfSONAR 
protocol we do not see another possibility than to try a solution 
outside of perfSONAR (by directly writing new HADES data into the CNM 
database). This situation is certainly at the upper-end of the data 
volume to be transferred via the perfSONAR protocols, but it would 
nevertheless be worthwhile to be addressed by it. In addition, the 
performance of large volume RRD MA requests has also been subject to 
discussion for a longer period of time so that we think that a 
modification to the perfSONAR protocol may be considered.


The difficulties are more or less tied to the flexibility that is 
foreseen in the NMWG format as briefly summarized in the following. 
There is the concept that data and metadata contain pointers so that a 
message typically looks like

“Metadata2, metadata1 (containing reference to metadata2), data 
(containing reference to metadata1)”,

but can also be organized in a different arbitrary manner. A further 
possibility is the potential multiple use of references (for filter 
metadata) which may look as follows.

“Filter metadata, metadata1 (containing reference to filter metadata), 
metadata2 (containing reference to filter metadata), data1 (containing 
reference to metadata1), data2 (containing reference to metadata2)”

The processing would therefore require an interleaved referencing to 
previous metadata so that a serial processing of data is not possible 
and requires that a complicated XML parsing tree is built in memory. 
Even though the latter case has currently not been used yet (to our 
knowledge), we have to address the amount of data to be kept in memory 
during the XML parsing.


What should be done is to try to allow for a serialized processing. 
There could be a metadata flag that says that the data in the message 
allow for serialized processing. This means that possibilities for 
arbitrary ordering are not used in this case and also that some metadata 
are repeated (e.g. the filtering metadata mentioned above). The idea of 
the flag is that services which have no need for high-volume data 
exchange may ignore the flag and process data as before so that there is 
no need for changes. Other services would be required to have a new 
library for parsing. We have to check for potential problems if there 
are messages suitable for stream processing and others are not stream 
processing-enabled (e.g. one RRD MA instance sends stream 
processing-enabled messages and the others not).

A further alternative would be a RESTful architecture which could make 
use of a modified MetaDataKeyRequest. A client asks for a key for a 
certain parameter set. The key in the answer is then a URL which is the 
location where the client can fetch the data together with a metadata 
description. The further data transfer is then done with HTTP which 
allows for stream processing. The data would not be wrapped in XML 
anymore which would minimize the overhead. However, this solution would 
require larger modifications to perfSONAR.

Best regards

Verena and Andreas

@Szymon: Could you please add this to the agenda of the perfSONAR 
meeting, preferable on Wednesday?

--
Dr. Andreas Hanemann      
DFN-Verein
Stresemannstr. 78
10963 Berlin, Germany
Tel.:   ++49 30 884299 64
Fax:    ++49 30 884299 70
E-Mail:  (Please do 
not use  anymore)