All things related to multicast

["Marshall Eubanks" <>]

Craig;

I am going to bring L3 into this. There seems to be a real problem at the
L3 / I2 multicast peering at Nasa AMES -

symptom - multicast data comes (L3 into I2) for about 5 seconds, dies, and 
then
revives after 3 minutes, only to repear the cycle.

This sounds like something is messed up with PIM keep alive messages to me.

Marshall 


On Thu, 4 Nov 2004 15:34:19 -0500
 Matthew Davy 
<>
 wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I'm looking at this right now.  I'm seeing the same behavior here at 
> IU.  I see good, consistent PIM state all the way back to the 
> Abilene/Level3 peering at NASA Ames.  Traffic comes for 5 seconds then 
> stops for a long while, 3-5 minutes maybe.  But when it stops, it stops 
> even at the first Abilene interfaces.   So it must be getting hung up 
> somewhere before there.
> 
> We just changed our setup at the Ames MIX to use a GIGE over MPLS 
> thing.  I suspect the PIM joins or multicast packets might be getting 
> lost on that link.
> 
> Okay, I just changed the preference on the IU peering with Level3 to 
> prefer the direct path to Level3 over the path through Abilene then to 
> Level3.  This resolved the problem which really makes me suspect that 
> Ethernet over MPLS link to NASA Ames.  We'll look into this.
> 
> - - Matt
> 
> Matthew Davy
> Chief Network Engineer, Indiana University
> University Information Technology Services / Abilene Network Operations 
> Center
> 2711 East 10th Street, Bloomington IN, 47403
> 
>  /  812.855.7728
> PGP key fingerprint: A84D DFB6 9DD5 BEB4 1EF7  D713 956F F85C 6422 CBEB
> 
> 
> On Nov 4, 2004, at 1:03 PM, Marshall Eubanks wrote:
> 
> > I am going to take the liberty of forwarding this onto this list :
> >
> > Symptoms : from Nodak.edu, he can join an americafree.tv group. It dies
> > after 5 seconds or so, and comes back after 3 minutes.
> >
> > The timing makes me think of PIM softstate intervals, but nothing 
> > specifc comes to
> > mind at present. It's not caused by either the source or the receiver. 
> > He says
> > he can get other sources (not from here) just fine.
> >
> > Can someone else report whether or not they see this ? Any ideas ?
> >
> > Regards
> > Marshall
> >
> >
> > --- the forwarded message follows ---
> >
> > From: "Bruce Curtis" 
> > <>
> > Date: November 4, 2004 12:11:40 PM EST
> > To: "Marshall Eubanks" 
> > <>
> > Cc: 
> > ,
> >  "Marshall Eubanks" 
> > <>,
> >  "John M Hicks" 
> > <>,
> >  
> > ,
> >  
> > 
> > Subject: Re: multicast issue = 5828
> > Reply-To: 
> > 
> >
> >
> >
> > On Thu, November 4, 2004 8:24 am, Marshall Eubanks said:
> >> There are several possibilities. What OS / player are you using.
> >
> >   I'm using Quicktime in MAC OS X.  I also used Quicktime under Windows
> > with the same result.  Both had worked fine to view this stream
> > previously.
> >
> >>  Quicktime, for example, had a bug a good while ago that meant that it
> >> would not
> >> refresh IGMP member messages (or maybe it was the Mac stack, but that 
> >> was
> >> the effect).
> >> So, groups will be joined but would then time out and die.
> >>
> >> What I was trying to make sure was that the problem was not in the
> >> receiver computer -
> >> thus the request to use rtpdump. (rtpqual uses the same code, so it 
> >> means
> >> that the problem is not in
> >> the player).
> >
> >   Right.
> >
> >>
> >> The times to stop you are talking about are much too long to be due to
> >> passing
> >> encapsulated data but rejecting multicast, so it has to be either a 
> >> data
> >> stoppage or a local problem
> >> (which includes the player, the player host, and your local DR).
> >>
> >> The next step would be to use tcpdump to see if IGMP membership 
> >> messages
> >> are being sent as they
> >> should be.
> >
> >   I'm out of town today but I have checked the router and it shows the
> > correct state indicating that my host is sending IGMP membership
> > messages correctly.  It works fine for other streams.
> >
> >
> >>
> >> Also, you could look in your DR AFTER the data stops. If the problem 
> >> is
> >> from the receiver, the group
> >> will be missing. If it is due to a data stoppage, then the group state
> >> will still be there, as
> >> the DR will still be receiving IGMP member messages from the group.
> >
> >   Multicast state is still there and correct in the DR and in the 
> > routers
> > in the path towards the source, including the Abilene core routers.
> >
> >   A "show ip mroute x count" on my Cisco router that peers with the
> > Northern Lights GigaPOP shows that traffc has stopped and is not
> > reaching our network.
> >
> >   Looking at the Abilene core routers they show 0 pps and 0 Kbps but 
> > have
> > the correct multicast state, indicating that the traffic stops before
> > reaching the Abilene backbone.
> >
> >>
> >> (Actually, quicktime sends RTCP traffic from each receiver, so that 
> >> should
> >> be visible going out
> >> bound if you can use quicktime.)
> >>
> >> Thanks
> >> Marshall
> >>
> >>
> >> On Wed, 3 Nov 2004 22:32:42 -0600 (CST)
> >>  "Bruce Curtis" 
> >> <>
> >>  wrote:
> >>>
> >>> On Wed, November 3, 2004 9:53 pm, Marshall Eubanks said:
> >>>> On Wed, 3 Nov 2004 15:45:35 -0600
> >>>>  Bruce Curtis 
> >>>> <>
> >>>>  wrote:
> >>>>>
> >>>>
> >>>> Yesterday I got all of my Abilene routes back.
> >>>>
> >>>>
> >>>>>    Today I'm receiving MBGP routes for 63.105.122.28 so the 
> >>>>> multicast
> >>>>> session starts, but now we are back to the other problem that I 
> >>>>> only
> >>>>> receive a few packets and then the session stops.
> >>>>>
> >>>>
> >>>> Get a copy of rtpdump and issue
> >>>>
> >>>> rtpdump -F ascii  233.64.133.120/8022
> >>>>
> >>>> before you start the player up.
> >>>>
> >>>> You should see a flood of packets come through. If the
> >>>> pkayer _still_ stops, then whether not rtpdump stops will tell
> >>>> whether this a player issue or a network issue.
> >>>>
> >>>> Marshall
> >>>
> >>>   The abilene core routers show 0 Kbps and 0 pps after the initial 
> >>> flow.
> >>>
> >>>
> >>>   I used rtpqual, I got the output below fairly quickly and then 
> >>> stopped
> >>> receiving packets.  Earlier I also verified with tcpdump that it was 
> >>> not
> >>> a problem with the viewer, I could see that the packets stopped.  
> >>> Also
> >>> the viewer works fine with other streams.
> >>>
> >>> $ rtpqual 233.64.133.120 8022
> >>> Defaulting to: rtpqual 233.64.133.120 8022 rtp
> >>> Report from: rtpqual 233.64.133.120 8022 rtp at Wed Nov  3 22:25:36 
> >>> 2004
> >>> T Pkts Loss  % Late Bytes |  Pkts  Loss  %  Late     kB Sender
> >>> 36  44   2   4   1  46366 |    44    2   4    1      45 63.105.122.28
> >>> 37  29   2   6   0  29195 |    73    4   5    1      73 63.105.122.28
> >>> 38  28   3   9   0  27462 |   101    7   6    1     100 63.105.122.28
> >>> 39  30   2   6   0  31166 |   131    9   6    1     131 63.105.122.28
> >>> 40  31   0   0   0  30988 |   162    9   5    1     161 63.105.122.28
> >>>
> >>>   As the time stamp shows there was a three minute pause and then I
> >>> received another group of packets.
> >>>
> >>> 41   9   0   0   0   8650 |   171    9   5    1     169 63.105.122.28
> >>> Report from: rtpqual 233.64.133.120 8022 rtp at Wed Nov  3 22:28:32 
> >>> 2004
> >>> T Pkts Loss  % Late Bytes |  Pkts  Loss  %  Late     kB Sender
> >>> 31   4  12  75   0   4185 |   175   21  10    1     173 63.105.122.28
> >>> 32  40   2   4   0  44018 |   215   23   9    1     216 63.105.122.28
> >>> 33  34   0   0   0  40881 |   249   23   8    1     256 63.105.122.28
> >>> 34  40   0   0   0  41966 |   289   23   7    1     297 63.105.122.28
> >>> 35  40   0   0   0  44122 |   329   23   6    1     340 63.105.122.28
> >>> 36  36   0   0   0  39468 |   365   23   5    1     379 63.105.122.28
> >>> 37  33   0   0   0  37322 |   398   23   5    1     415 63.105.122.28
> >>>
> >>>   And then is stopped again, will likely have another burst in 3
> >>> minutes...
> >>>
> >>>   Today I looked at the whole list of MBGP routes from level 3 and 
> >>> found
> >>> another site with an MSDP entry that came through level 3.  I used
> >>> rtpqual for that site and it worked fine.  But it looked like that 
> >>> site
> >>> was in Chicago if I remember correctly.
> >>>
> >>>
> >>>
> >>>
> >>>>
> >>>>
> >>>>
> >>>>>
> >>>>> On Tuesday, November 2, 2004, at 01:25  PM, Bruce Curtis wrote:
> >>>>>
> >>>>>>
> >>>>>> On Tuesday, November 2, 2004, at 01:14  PM, John M Hicks wrote:
> >>>>>>
> >>>>>>> Bruce,
> >>>>>>> Back at it again today.  Can you send me a snap shot of the
> >>> multicast
> >>>>>>> state
> >>>>>>> for what you are looking at?  Also, where are you getting the SAs
> >>>>>>> from?  Do
> >>>>>>> you have a level3 peering?
> >>>>>>> Thanks,
> >>>>>>> -john
> >>>>>>
> >>>>>>   We don't have a level 3 peering, but the source 63.105.122.28 is
> >>> on
> >>>>>> level 3.  Our only multicast peering is with the Northern Lights
> >>>>>> GigaPOP.
> >>>>>>
> >>>>>>   The problem is different that it was last week.  Now I get no
> >>>>>> multicast packets from  63.105.122.28 for group 233.64.133.110,
> >>> which
> >>>>>> makes sense since I'm not receiving any routes for 63.105.122.28 
> >>>>>> in
> >>>>>> MBGP anymore, and so there is an rpf failure.  The email I sent
> >>>>>> yesterday from that included the entries from the Level 3 looking
> >>>>>> glass showed that there is a problem within the Level 3 network 
> >>>>>> and
> >>>>>> the MBGP announcements for 63.105.122.28 aren't making it to the
> >>> West
> >>>>>> Coast portion of Level 3's network.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> i2.ndsu>show ip mroute 233.64.133.110
> >>>>>> IP Multicast Routing Table
> >>>>>> Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C -
> >>>>>> Connected,
> >>>>>>        L - Local, P - Pruned, R - RP-bit set, F - Register flag,
> >>>>>>        T - SPT-bit set, J - Join SPT, M - MSDP created entry,
> >>>>>>        X - Proxy Join Timer Running, A - Candidate for MSDP
> >>>>>> Advertisement,
> >>>>>>        U - URD, I - Received Source Specific Host Report, Z -
> >>>>>> Multicast Tunnel
> >>>>>>        Y - Joined MDT-data group, y - Sending to MDT-data group
> >>>>>> Outgoing interface flags: H - Hardware switched
> >>>>>>  Timers: Uptime/Expires
> >>>>>>  Interface state: Interface, Next-Hop or VCD, State/Mode
> >>>>>>
> >>>>>> (*, 233.64.133.110), 00:00:50/stopped, RP 134.129.65.254, flags: 
> >>>>>> SP
> >>>>>>   Incoming interface: Null, RPF nbr 0.0.0.0
> >>>>>>   Outgoing interface list: Null
> >>>>>>
> >>>>>> (63.105.122.28, 233.64.133.110), 00:00:50/00:02:40, flags: P
> >>>>>>   Incoming interface: FastEthernet1/0/0, RPF nbr 134.129.107.3
> >>>>>>   Outgoing interface list: Null
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> i2.ndsu>show ip mbgp 63.105.122.28
> >>>>>> % Network not in table
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 2.ndsu>show ip msdp sa 233.64.133.110
> >>>>>> MSDP Source-Active Cache - 2 entries for 233.64.133.110
> >>>>>> (63.105.122.28, 233.64.133.110), RP 206.61.163.252, MBGP/AS 1239,
> >>>>>> 1w3d/00:05:13, Peer 192.42.152.174
> >>>>>>
> >>>>>>   Even though whois shows that 63.105.122.28 belongs to uunet and
> >>>>>> 206.61.163.252 belongs to sprint traceroutes show that both are on
> >>>>>> Level 3's network.
> >>>>>>
> >>>>>>
> >>>>>> traceroute 206.61.163.252
> >>>>>> traceroute to 206.61.163.252 (206.61.163.252), 30 hops max, 40 
> >>>>>> byte
> >>>>>> packets
> >>>>>>  1  a095.not-a-bridge.ndsu.nodak.edu (134.129.95.100)  0.859 ms
> >>> 0.779
> >>>>>> ms  0.321 ms
> >>>>>>  2  fast100.i1.ndsu.nodak.edu (134.129.107.3)  0.647 ms  0.444 ms
> >>>>>> 0.576 ms
> >>>>>>  3  router.gig.hecn.ndsu.nodak.edu (134.129.29.41)  1.166 ms  
> >>>>>> 1.136
> >>> ms
> >>>>>>  0.739 ms
> >>>>>>  4  165.234.165.131 (165.234.165.131)  1.634 ms  1.004 ms  1.287 
> >>>>>> ms
> >>>>>>  5  165.234.165.129 (165.234.165.129)  1.748 ms  1.677 ms  1.833 
> >>>>>> ms
> >>>>>>  6  sl-gw33-chi-0-0.sprintlink.net (144.223.34.197)  37.837 ms
> >>> 38.19
> >>>>>> ms  41.226 ms
> >>>>>>  7  sl-bb22-chi-4-0.sprintlink.net (144.232.26.21)  38.921 ms
> >>> 40.394
> >>>>>> ms 206.61 38.611 ms
> >>>>>>  8  sl-st21-chi-13-0.sprintlink.net (144.232.20.91)  39.264 ms
> >>> 39.245
> >>>>>> ms  41.346 ms
> >>>>>>  9  sl-st20-chi-1-0.sprintlink.net (144.232.8.102)  39.151 ms
> >>> 40.284
> >>>>>> ms  39.616 ms
> >>>>>> 10  so-2-1-0.edge1.chicago1.level3.net (209.0.225.21)  39.285 ms
> >>>>>> 40.818 ms  38.925 ms
> >>>>>> 11  so-2-1-0.bbr2.chicago1.level3.net (209.244.8.13)  51.131 ms
> >>>>>> 88.688 ms  40.646 ms
> >>>>>> 12  ge-0-3-0.bbr2.washington1.level3.net (64.159.0.229)  58.12 ms
> >>>>>> so-2-0-0.bbr2.washington1.level3.net (209.247.10.130)  58.892 ms
> >>>>>> 61.364 ms
> >>>>>> 13  ge-7-1.ipcolo1.washington1.level3.net (4.68.121.75)  58.171 ms
> >>>>>> ge-7-0.ipcolo1.washington1.level3.net (4.68.121.11)  58.313 ms
> >>>>>> ge-9-1.ipcolo1.washington1.level3.net (4.68.121.107)  58.163 ms
> >>>>>> 14  unknown.level3.net (63.210.25.154)  59.39 ms^C^\Quit
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> Bruce Curtis                           
> >>>>>> 
> >>>>>> Certified NetAnalyst II                                701-231-8527
> >>>>>> North Dakota State University
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> ---
> >>>>> Bruce Curtis                            
> >>>>> 
> >>>>> Certified NetAnalyst II                         701-231-8527
> >>>>> North Dakota State University
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> ---
> >>> Bruce Curtis                             
> >>> 
> >>> Certified NetAnalyst II                                701-231-8527
> >>> North Dakota State University
> >>>
> >>
> >>
> >
> >
> > ---
> > Bruce Curtis                             
> > 
> > Certified NetAnalyst II                                701-231-8527
> > North Dakota State University
> >
> >
> >
> >
> Matthew Davy
> Chief Network Engineer, Indiana University
> University Information Technology Services / Abilene Network Operations 
> Center
> 2711 East 10th Street, Bloomington IN, 47403
> 
>  /  812.855.7728
> PGP key fingerprint: A84D DFB6 9DD5 BEB4 1EF7  D713 956F F85C 6422 CBEB
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.3 (Darwin)
> 
> iD8DBQFBipJKlW/4XGQiy+sRAl1vAKCpGrfWX0xjMAVC2qczuOb5XrZ42wCg0M13
> W8VwCT0C2Q35av2Zg5mtA6U=
> =6Gc+
> -----END PGP SIGNATURE-----
>