All things related to multicast

Colleagues,

This issue has been resolved!

Here is a post-mortem.

I would like to acknowledge the specific contributions of the Internet2 
Multicast Working Group members Dave Farmer, [University of Minnesota] Alan 
Crosswell [Columbia University], Tim Shortall [University of Maryland], Dave 
Devereaux-Weber, [University of Wisconsin-Madison] for their assistance on 
this issue. 

Assistance was also provided by Randy Winchester [Massachusetts Institute of 
Technology], Eric Fisher [Embry Riddle Aeronautical University], Rich Griffin 
[OSTN], Jamie Maxwell [University of Montana], Amr Mubarak  [University of 
Nevada, Las Vegas], Bob Riddle [Internet2], Dave Bristol [Microsoft]. 

Problem Statement
Case Western Reserve University was facing quality problems [packet loss, 
packet misprioritization] on their OSTN IP Television multicast streams. 
Network engineers at Case Western Reserve University Information Technology 
Services performed a number of tests but were initially unsuccessful in 
identifying the root cause of the problem. Time was short, back-to-school 
style.

Solution Process
Based on the immediate feedback [which came within 24hr of my post] from the 
Internet2 Multicast Working Group call for help that I posted On Thursday 
September 1st, the OSTN and Case Western Reserve University engineering teams 
tried many permutations and combinations of solutions including:
 
1) Based on Dave's [Wisconsin] advice we looked at the router and swith ports 
directly to see if dropped packets are reported. 
2) Based on Dave's [Wisconsin] advice we used iperf to measure jitter.
3) Based on Dave's [Wisconsin] advice we analyzed the difference between 
client packet loss and delay and network packet loss and delay.
4) Based on Tim's [UMD] advice configured cgmp on the layer 3 interfaces
5) Based on Tim's [UMD] advice opened ports in the layer2 firewalls between 
the servers and the clients
6) Based on Dave's [UMN] advice we measured packet collision and network 
congestion. 
7) Based on Dave's [UMN] advice we verified that we were using PIM
Sparse Mode and verified that there was no duplex mismatch. 

Also, engineers from MIT, CASE, ERAU, I2, OSTN, UMT, and Microsoft 
collaborated on numerous conference calls to brainstorm and to conduct 
testing.  

Solution
In addition to implementing the advice outlined above, the CASE and OSTN 
engineering teams:
1) Applied ToS tags to prioritize video packets.
2) Created a private network was created between the encoders and 
distribution servers to optimize the route between distribution servers and 
layer2/3 network infrastructure.
3) Reconfigured the multicast tree for more efficient packet delivery.

Outcome
As of Thursday September 15th, Case Western Reserve University has been 
providing OSTN IP Television as a production multicast service to 17,000 
ports on their IP network just in time for their 9,000 incoming students for 
the Fall 2005 semester. 
 
The effort underscores the value of the Internet2 Multicast Working Group and 
collaboration in higher education.

Thanks for all of your efforts.
Prashant Chopra
Chief Architect
CampusEAI Consortium