Skip to Content.
Sympa Menu

perfsonar-dev - Re: [pS-dev] Efficient large Data transfer

Subject: perfsonar development work

List archive

Re: [pS-dev] Efficient large Data transfer


Chronological Thread 
  • From: Jochen Reinwand <>
  • To: Joe Metzger <>
  • Cc: maxim <>, "'Nicolas Simar'" <>, "'Nina Jeliazkova'" <>, "'schmitz'" <>, "'WiN-Labor'" <>,
  • Subject: Re: [pS-dev] Efficient large Data transfer
  • Date: Wed, 30 Apr 2008 09:09:01 +0200
  • Organization: DFN Verein

We already discussed this in Zagreb and I don't see the need to start the
discussion again. We agreed with Martin that this is the best way to proceed.

Jochen

On Tuesday 29 April 2008 17:12, Joe Metzger wrote:
> Jochen,
> I realize that you need to work out a solution that will allow you to
> meet your commitments. But I am concerned that this is a significant
> deviation from the current perfSONAR standards in both the protocol
> used for the transport, and the format of the data exchanged.
>
> Is it necessary to change both the transport protocol (rsync), and the
> data storage format (Compressed perl Storables) to achieve your
> performance
> objectives?
>
> Do you think we need to incorporate "external" transport and/or data
> formats
> into the perfSONAR standards?
>
> Does this data exchange need to be considered part of the perfSONAR
> framework? Or, could
> it be considered as a 'backed' interface used by a custom measurement
> system (hades)
> to push data into a perfSONAR Measurement Archive. (Similar to the
> way several
> people are using rsync to update the rrd files served by the rrdma.)
>
>
> --Joe
>
> On Apr 29, 2008, at 9:48 AM, Jochen Reinwand wrote:
> > On Tuesday 29 April 2008 16:27, maxim wrote:
> >> and what "perl binary format" means ? serialized perl data
> >> structures with
> >> Storable ?
> >
> > Our data on the central server is stored in a directory per day and
> > in there
> > one file for each measurement. The file is created this way:
> >
> > use Storable qw(nstore);
> > use Compress::Zlib;
> > my $frozen = nfreeze($ref);
> > my $compressed = compress($frozen);
> > nstore \$compressed, $file;
> >
> > I set up an rsync daemon serving our data directory. Via perfSONAR
> > it will be
> > possible to get the URL for the corresponding file.
> >
> > Jochen
> >
> > --
> > Jochen Reinwand, DFN-Labor
> > Friedrich-Alexander-Universität Erlangen-Nürnberg
> > Regionales RechenZentrum Erlangen (RRZE)
> > Martensstraße 1, 91058 Erlangen, Germany
> > Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941
> >
> > www.win-labor.dfn.de

--
Jochen Reinwand, DFN-Labor
Friedrich-Alexander-Universität Erlangen-Nürnberg
Regionales RechenZentrum Erlangen (RRZE)
Martensstraße 1, 91058 Erlangen, Germany
Tel. +49 9131 85-28689, -28800, Fax +49 9131 302941

www.win-labor.dfn.de



Archive powered by MHonArc 2.6.16.

Top of Page