All things related to multicast

["Dale W. Carder" <>]

My first thought would be to set up a standalone, dedicated NNTP 
infrastructure.  

Maybe that is too old-school.  Otherwise I second looking into
bittorrent.

Dale


On Feb 21, 2013, at 7:24 PM, Steven Wallace wrote:

> 
> On Feb 21, 2013, at 8:23 PM, Steven Wallace 
> <>
>  wrote:
> 
>> Here are some random thoughts:
>> 
>> - consider using bittorrent vs. multicast
>> - encapsulate files (or groups of file) similar to PGP e-mail (i.e., 
>> multiple public-key encrypted copies of the session key).
>> 
>> 
>> 
>> On Feb 21, 2013, at 8:06 PM, Michael Laufer 
>> <>
>>  wrote:
>> 
>>> I recently joined this working group and would like some advice.
>>> 
>>> My organization is considering the possibility of using new methodologies 
>>> to distribute near real time satellite weather data streams to 
>>> international & domestic partners/users, probably via Internet2 and 
>>> international peers. The previous generation of satellites would produce 
>>> ~30 Gbytes/day of data products but the new generation produces 3-4 
>>> Terabytes/day. Existing distribution methods will not economically scale, 
>>> especially internationally. The data streams are all files and any packet 
>>> loss would cause a file to become corrupted and unusable.
>>> 
>>> We would like to investigate the possible use of multicasting to 
>>> implement this. To do this we would need a very large file distribution 
>>> capability. I have been researching some possibilities but would like 
>>> your suggestions on what software/applications/systems could be used for 
>>> this by us and our partners/users, especially with our specific 
>>> requirements. We would also welcome any other suggested methodologies 
>>> that could achieve similar results.
>>> 
>>> The following are some details that may be useful:
>>> This information is for one satellite only (in a polar orbit). In a few 
>>> years we may need multiples of this.
>>> Number of source distribution sites: 1 (with a possible backup)
>>> Number of destination partners/users: 10-30 (may grow if this is 
>>> successful but not > 100)
>>> Each file has a companion checksum type file (a few bytes only) 
>>> associated with it. The numbers of files listed below do not include 
>>> these checksum files.
>>> Number of products streams: ~100+. Partners/users must be able to 
>>> subscribe to individual streams, some may want all steams, others subsets.
>>> Number of files/product stream: 2 - ~200, most streams with large files 
>>> have ~70. All files in a stream are ~same size.
>>> How often to distribute: Every ~100 minutes (~14 times/day).
>>> Data per distribution: ~250 Gbytes/orbit (~3.5 Tbytes/day)
>>> Files per distribution: ~11000/orbit (~150000/day)
>>> Number of files (approximate) per file size:
>>> 1500 < 10 Kbytes < 2500 < 100 KBytes < 1000 < 1 Mbyte < 1500 < 10 Mbytes 
>>> < 4000 < 100 Mbytes < 500 < ~600 Mbytes
>>> Time for distribution: Goal is <= 10 minutes (10% of orbit). Can tolerate 
>>> up to ~50 minutes (50% of orbit). [Cannot be longer as need to allow 
>>> catch up time for missed orbits.]
>>> Bandwidth required: ~3+ Gbits/sec for ~10 minute distribution.
>>> Current I2 connection: 2 x 5 Gbits/sec (We may be able to add more if 
>>> needed).
>>> Security: UDP would security easier. Must be able to permit/deny each 
>>> separate partner/user request. Prefer separate sending system and any 
>>> return/feedback system (for missed files/retransmission notification).
>>> Time frame: ~ Summer to start initial testing.
>>> 
>>> We would start initially with some of the smaller product streams and add 
>>> additional larger streams as partners/users request. Would probably start 
>>> and test with domestic partners/user directly on I2.
>>> 
>>> We have discussed bandwidth needs with I2 and GÈANT and they will support 
>>> this effort.
>>> 
>>> Please NOTE however: THIS IS ONLY EXPLORATORY AND NOT ANY DECISION OR 
>>> COMMITMENT TO IMPLEMENT THIS DATA DISTRIBUTION!!!!!
>>> 
>>> Thanks in advance for any help and/or suggestions you can offer.
>>> 
>>> Michael
>>> 
>>> -- 
>>> Michael Laufer
>>> NOAA/NESDIS/OSD/GSD Systems Engineering - Network & Security Architect
>>> Contractor, Columbus Technologies & Services
>>> NOAA - NSOF 4231 Suitland Road Suitland MD 20746
>>> Office: (301) 817-4410 Mobile: (301) 340-8772
>>> Note: I am not a government employee and have no legal authority to 
>>> obligate any federal, state, or local government to perform any action or 
>>> payment.
>>> 
>> 
>