Skip to Content.
Sympa Menu

perfsonar-dev - Re: [pS-dev] Efficient large Data transfer

Subject: perfsonar development work

List archive

Re: [pS-dev] Efficient large Data transfer


Chronological Thread 
  • From: Joe Metzger <>
  • To: Jochen Reinwand <>
  • Cc: maxim <>, "'Nicolas Simar'" <>, "'Nina Jeliazkova'" <>, "'schmitz'" <>, "'WiN-Labor'" <>,
  • Subject: Re: [pS-dev] Efficient large Data transfer
  • Date: Tue, 29 Apr 2008 10:12:06 -0500

Jochen,
I realize that you need to work out a solution that will allow you to
meet your commitments. But I am concerned that this is a significant
deviation from the current perfSONAR standards in both the protocol
used for the transport, and the format of the data exchanged.

Is it necessary to change both the transport protocol (rsync), and the
data storage format (Compressed perl Storables) to achieve your performance
objectives?

Do you think we need to incorporate "external" transport and/or data formats
into the perfSONAR standards?

Does this data exchange need to be considered part of the perfSONAR framework? Or, could
it be considered as a 'backed' interface used by a custom measurement system (hades)
to push data into a perfSONAR Measurement Archive. (Similar to the way several
people are using rsync to update the rrd files served by the rrdma.)


--Joe



On Apr 29, 2008, at 9:48 AM, Jochen Reinwand wrote:

On Tuesday 29 April 2008 16:27, maxim wrote:
and what "perl binary format" means ? serialized perl data structures with
Storable ?

Our data on the central server is stored in a directory per day and in there
one file for each measurement. The file is created this way:

use Storable qw(nstore);
use Compress::Zlib;
my $frozen = nfreeze($ref);
my $compressed = compress($frozen);
nstore \$compressed, $file;

I set up an rsync daemon serving our data directory. Via perfSONAR it will be
possible to get the URL for the corresponding file.

Jochen

--
Jochen Reinwand, DFN-Labor
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales RechenZentrum Erlangen (RRZE)
Martensstraße 1, 91058 Erlangen, Germany
Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941

www.win-labor.dfn.de





Archive powered by MHonArc 2.6.16.

Top of Page