Skip to Content.
Sympa Menu

perfsonar-dev - Re: [pS-dev] Efficient large Data transfer

Subject: perfsonar development work

List archive

Re: [pS-dev] Efficient large Data transfer


Chronological Thread 
  • From: "Jeff W. Boote" <>
  • To: Jochen Reinwand <>
  • Cc: Joe Metzger <>, maxim <>, "'Nicolas Simar'" <>, "'Nina Jeliazkova'" <>, "'schmitz'" <>, "'WiN-Labor'" <>,
  • Subject: Re: [pS-dev] Efficient large Data transfer
  • Date: Wed, 30 Apr 2008 17:47:28 -0600

There are far more people in the perfSONAR community than were at Zagreb.

It would be good to explain what it is you are doing, and why for the full community.

jeff

On Apr 30, 2008, at 1:09 AM, Jochen Reinwand wrote:

We already discussed this in Zagreb and I don't see the need to start the
discussion again. We agreed with Martin that this is the best way to proceed.

Jochen

On Tuesday 29 April 2008 17:12, Joe Metzger wrote:
Jochen,
I realize that you need to work out a solution that will allow you to
meet your commitments. But I am concerned that this is a significant
deviation from the current perfSONAR standards in both the protocol
used for the transport, and the format of the data exchanged.

Is it necessary to change both the transport protocol (rsync), and the
data storage format (Compressed perl Storables) to achieve your
performance
objectives?

Do you think we need to incorporate "external" transport and/or data
formats
into the perfSONAR standards?

Does this data exchange need to be considered part of the perfSONAR
framework? Or, could
it be considered as a 'backed' interface used by a custom measurement
system (hades)
to push data into a perfSONAR Measurement Archive. (Similar to the
way several
people are using rsync to update the rrd files served by the rrdma.)


--Joe

On Apr 29, 2008, at 9:48 AM, Jochen Reinwand wrote:
On Tuesday 29 April 2008 16:27, maxim wrote:
and what "perl binary format" means ? serialized perl data
structures with
Storable ?

Our data on the central server is stored in a directory per day and
in there
one file for each measurement. The file is created this way:

use Storable qw(nstore);
use Compress::Zlib;
my $frozen = nfreeze($ref);
my $compressed = compress($frozen);
nstore \$compressed, $file;

I set up an rsync daemon serving our data directory. Via perfSONAR
it will be
possible to get the URL for the corresponding file.

Jochen

--
Jochen Reinwand, DFN-Labor
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales RechenZentrum Erlangen (RRZE)
Martensstraße 1, 91058 Erlangen, Germany
Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941

www.win-labor.dfn.de

--
Jochen Reinwand, DFN-Labor
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales RechenZentrum Erlangen (RRZE)
Martensstraße 1, 91058 Erlangen, Germany
Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941

www.win-labor.dfn.de





Archive powered by MHonArc 2.6.16.

Top of Page