All things related to multicast

[Steven Wallace <>]

On Feb 21, 2013, at 8:23 PM, Steven Wallace <> wrote:

Here are some random thoughts:

- consider using bittorrent vs. multicast

- encapsulate files (or groups of file) similar to PGP e-mail (i.e., multiple public-key encrypted copies of the session key).

On Feb 21, 2013, at 8:06 PM, Michael Laufer <> wrote:

I recently joined this working group and would like some advice.

My organization is considering the possibility of using new methodologies to distribute near real time satellite weather data streams to international & domestic partners/users, probably via Internet2 and international peers. The previous generation of satellites would produce ~30 Gbytes/day of data products but the new generation produces 3-4 Terabytes/day. Existing distribution methods will not economically scale, especially internationally. The data streams are all files and any packet loss would cause a file to become corrupted and unusable.

We would like to investigate the possible use of multicasting to implement this. To do this we would need a very large file distribution capability. I have been researching some possibilities but would like your suggestions on what software/applications/systems could be used for this by us and our partners/users, especially with our specific requirements. We would also welcome any other suggested methodologies that could achieve similar results.

The following are some details that may be useful:
This information is for one satellite only (in a polar orbit). In a few years we may need multiples of this.
Number of source distribution sites: 1 (with a possible backup)
Number of destination partners/users: 10-30 (may grow if this is successful but not > 100)
Each file has a companion checksum type file (a few bytes only) associated with it. The numbers of files listed below do not include these checksum files.
Number of products streams: ~100+. Partners/users must be able to subscribe to individual streams, some may want all steams, others subsets.
Number of files/product stream: 2 - ~200, most streams with large files have ~70. All files in a stream are ~same size.
How often to distribute: Every ~100 minutes (~14 times/day).
Data per distribution: ~250 Gbytes/orbit (~3.5 Tbytes/day)
Files per distribution: ~11000/orbit (~150000/day)
Number of files (approximate) per file size:
1500 < 10 Kbytes < 2500 < 100 KBytes < 1000 < 1 Mbyte < 1500 < 10 Mbytes < 4000 < 100 Mbytes < 500 < ~600 Mbytes
Time for distribution: Goal is <= 10 minutes (10% of orbit). Can tolerate up to ~50 minutes (50% of orbit). [Cannot be longer as need to allow catch up time for missed orbits.]
Bandwidth required: ~3+ Gbits/sec for ~10 minute distribution.
Current I2 connection: 2 x 5 Gbits/sec (We may be able to add more if needed).
Security: UDP would security easier. Must be able to permit/deny each separate partner/user request. Prefer separate sending system and any return/feedback system (for missed files/retransmission notification).
Time frame: ~ Summer to start initial testing.

We would start initially with some of the smaller product streams and add additional larger streams as partners/users request. Would probably start and test with domestic partners/user directly on I2.

We have discussed bandwidth needs with I2 and GÈANT and they will support this effort.

Please NOTE however: THIS IS ONLY EXPLORATORY AND NOT ANY DECISION OR COMMITMENT TO IMPLEMENT THIS DATA DISTRIBUTION!!!!!

Thanks in advance for any help and/or suggestions you can offer.

Michael

-- 
Michael Laufer
NOAA/NESDIS/OSD/GSD Systems Engineering - Network & Security Architect
Contractor, Columbus Technologies & Services
NOAA - NSOF 4231 Suitland Road Suitland MD 20746
Office: (301) 817-4410 Mobile: (301) 340-8772
Note: I am not a government employee and have no legal authority to obligate any federal, state, or local government to perform any action or payment.