Transport protocols and bulk file transfer

[Bill Owens <>]

Transport folks, this note came to the Multicast working group yesterday, but 
after a little bit of digestion the overall opinion was that multicast 
probably wouldn't cut it. I wonder if there might be better answers to be 
found here?

Bill.

On Thu, Feb 21, 2013 at 08:06:38PM -0500, Michael Laufer wrote:
>    I recently joined this working group and would like some advice.
> 
>    My organization is considering the possibility of using new methodologies
>    to distribute near real time satellite weather data streams to
>    international & domestic partners/users, probably via Internet2 and
>    international peers. The previous generation of satellites would produce
>    ~30 Gbytes/day of data products but the new generation produces 3-4
>    Terabytes/day. Existing distribution methods will not economically scale,
>    especially internationally. The data streams are all files and any packet
>    loss would cause a file to become corrupted and unusable.
> 
>    We would like to investigate the possible use of multicasting to 
> implement
>    this. To do this we would need a very large file distribution capability.
>    I have been researching some possibilities but would like your 
> suggestions
>    on what software/applications/systems could be used for this by us and 
> our
>    partners/users, especially with our specific requirements. We would also
>    welcome any other suggested methodologies that could achieve similar
>    results.
> 
>    The following are some details that may be useful:
>    This information is for one satellite only (in a polar orbit). In a few
>    years we may need multiples of this.
>    Number of source distribution sites: 1 (with a possible backup)
>    Number of destination partners/users: 10-30 (may grow if this is
>    successful but not > 100)
>    Each file has a companion checksum type file (a few bytes only) 
> associated
>    with it. The numbers of files listed below do not include these checksum
>    files.
>    Number of products streams: ~100+. Partners/users must be able to
>    subscribe to individual streams, some may want all steams, others 
> subsets.
>    Number of files/product stream: 2 - ~200, most streams with large files
>    have ~70. All files in a stream are ~same size.
>    How often to distribute: Every ~100 minutes (~14 times/day).
>    Data per distribution: ~250 Gbytes/orbit (~3.5 Tbytes/day)
>    Files per distribution: ~11000/orbit (~150000/day)
>    Number of files (approximate) per file size:
>    1500 < 10 Kbytes < 2500 < 100 KBytes < 1000 < 1 Mbyte < 1500 < 10 Mbytes 
> <
>    4000 < 100 Mbytes < 500 < ~600 Mbytes
>    Time for distribution: Goal is <= 10 minutes (10% of orbit). Can tolerate
>    up to ~50 minutes (50% of orbit). [Cannot be longer as need to allow 
> catch
>    up time for missed orbits.]
>    Bandwidth required: ~3+ Gbits/sec for ~10 minute distribution.
>    Current I2 connection: 2 x 5 Gbits/sec (We may be able to add more if
>    needed).
>    Security: UDP would security easier. Must be able to permit/deny each
>    separate partner/user request. Prefer separate sending system and any
>    return/feedback system (for missed files/retransmission notification).
>    Time frame: ~ Summer to start initial testing.
> 
>    We would start initially with some of the smaller product streams and add
>    additional larger streams as partners/users request. Would probably start
>    and test with domestic partners/user  directly on I2.
> 
>    We have discussed bandwidth needs with I2 and GEANT and they will support
>    this effort.
>    Please NOTE however: THIS IS ONLY EXPLORATORY AND NOT ANY DECISION OR
>    COMMITMENT TO IMPLEMENT THIS DATA DISTRIBUTION!!!!!
>    Thanks in advance for any help and/or suggestions you can offer.
>    Michael
> 
>  --
>  Michael Laufer
>  NOAA/NESDIS/OSD/GSD Systems Engineering - Network & Security Architect
>  Contractor, Columbus Technologies & Services
>  NOAA - NSOF 4231 Suitland Road Suitland MD 20746
>  Office: (301) 817-4410 Mobile: (301) 340-8772
>  Note: I am not a government employee and have no legal authority to 
> obligate any federal, state, or local government to perform any action or 
> payment.