All things related to multicast

[Zenon Mousmoulas <>]

You should take a look at uftp, as it seems to be the one IP multicast based solution (that I know of) closer to your requirements:

http://www.tcnj.edu/~bush/uftp.html

+ It does retransmissions, like most of the approaches in this space, and some sort of congestion control.
+ It does symmetric encryption.
+ It does mutual authentication+authorization of senders and receivers.
- It might be difficult to get it to work the way you'd expect, as the sender is actually the client and the receiver is the server. You could perhaps write some wrappers and tools to accommodate a workflow of consecutive distributions.
- You could use file system paths to determine which servers get what files, however there is nothing to stop a server (that you do not control or trust) from receiving any and all files in a group. Of course you could split distribution in different groups in order to accommodate different access rights, but that would only go so far. If you are considering multicast and symmetric encryption, I think that admission control can not be as granular as you require.

Of course your biggest problem would be to get multicast to work consistently all the way to each user you want to reach (although there is some proxy in uftp that can help overcome individual issues). Taking that effort under consideration, perhaps you would be better off building an HTTP-based CDN :)

Regards,
Zenon Mousmoulas

On 22/02/2013 3:06 π.μ., Michael Laufer wrote:

I recently joined this working group and would like some advice.

My organization is considering the possibility of using new methodologies to distribute near real time satellite weather data streams to international & domestic partners/users, probably via Internet2 and international peers. The previous generation of satellites would produce ~30 Gbytes/day of data products but the new generation produces 3-4 Terabytes/day. Existing distribution methods will not economically scale, especially internationally. The data streams are all files and any packet loss would cause a file to become corrupted and unusable.

We would like to investigate the possible use of multicasting to implement this. To do this we would need a very large file distribution capability. I have been researching some possibilities but would like your suggestions on what software/applications/systems could be used for this by us and our partners/users, especially with our specific requirements. We would also welcome any other suggested methodologies that could achieve similar results.

The following are some details that may be useful:
This information is for one satellite only (in a polar orbit). In a few years we may need multiples of this.
Number of source distribution sites: 1 (with a possible backup)
Number of destination partners/users: 10-30 (may grow if this is successful but not > 100)
Each file has a companion checksum type file (a few bytes only) associated with it. The numbers of files listed below do not include these checksum files.
Number of products streams: ~100+. Partners/users must be able to subscribe to individual streams, some may want all steams, others subsets.
Number of files/product stream: 2 - ~200, most streams with large files have ~70. All files in a stream are ~same size.
How often to distribute: Every ~100 minutes (~14 times/day).
Data per distribution: ~250 Gbytes/orbit (~3.5 Tbytes/day)
Files per distribution: ~11000/orbit (~150000/day)
Number of files (approximate) per file size:
1500 < 10 Kbytes < 2500 < 100 KBytes < 1000 < 1 Mbyte < 1500 < 10 Mbytes < 4000 < 100 Mbytes < 500 < ~600 Mbytes
Time for distribution: Goal is <= 10 minutes (10% of orbit). Can tolerate up to ~50 minutes (50% of orbit). [Cannot be longer as need to allow catch up time for missed orbits.]
Bandwidth required: ~3+ Gbits/sec for ~10 minute distribution.
Current I2 connection: 2 x 5 Gbits/sec (We may be able to add more if needed).
Security: UDP would security easier. Must be able to permit/deny each separate partner/user request. Prefer separate sending system and any return/feedback system (for missed files/retransmission notification).
Time frame: ~ Summer to start initial testing.

We would start initially with some of the smaller product streams and add additional larger streams as partners/users request. Would probably start and test with domestic partners/user directly on I2.

We have discussed bandwidth needs with I2 and GÈANT and they will support this effort.

Please NOTE however: THIS IS ONLY EXPLORATORY AND NOT ANY DECISION OR COMMITMENT TO IMPLEMENT THIS DATA DISTRIBUTION!!!!!

Thanks in advance for any help and/or suggestions you can offer.

Michael

--
Network Applications Development
GRNET ‐ Greek Research & Technology Network S.A.

Mesogion 56,
GR‐11527 Athens
GREECE

tel: +30 2107474244
fax: +30 2107474490

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature