All things related to multicast

[Matthew Davy <>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm looking at this right now.  I'm seeing the same behavior here at 
IU.  I see good, consistent PIM state all the way back to the 
Abilene/Level3 peering at NASA Ames.  Traffic comes for 5 seconds then 
stops for a long while, 3-5 minutes maybe.  But when it stops, it stops 
even at the first Abilene interfaces.   So it must be getting hung up 
somewhere before there.

We just changed our setup at the Ames MIX to use a GIGE over MPLS 
thing.  I suspect the PIM joins or multicast packets might be getting 
lost on that link.

Okay, I just changed the preference on the IU peering with Level3 to 
prefer the direct path to Level3 over the path through Abilene then to 
Level3.  This resolved the problem which really makes me suspect that 
Ethernet over MPLS link to NASA Ames.  We'll look into this.

- - Matt

Matthew Davy
Chief Network Engineer, Indiana University
University Information Technology Services / Abilene Network Operations 
Center
2711 East 10th Street, Bloomington IN, 47403

 /  812.855.7728
PGP key fingerprint: A84D DFB6 9DD5 BEB4 1EF7  D713 956F F85C 6422 CBEB


On Nov 4, 2004, at 1:03 PM, Marshall Eubanks wrote:

> I am going to take the liberty of forwarding this onto this list :
>
> Symptoms : from Nodak.edu, he can join an americafree.tv group. It dies
> after 5 seconds or so, and comes back after 3 minutes.
>
> The timing makes me think of PIM softstate intervals, but nothing 
> specifc comes to
> mind at present. It's not caused by either the source or the receiver. 
> He says
> he can get other sources (not from here) just fine.
>
> Can someone else report whether or not they see this ? Any ideas ?
>
> Regards
> Marshall
>
>
> --- the forwarded message follows ---
>
> From: "Bruce Curtis" 
> <>
> Date: November 4, 2004 12:11:40 PM EST
> To: "Marshall Eubanks" 
> <>
> Cc: , "Marshall Eubanks" 
> <>, "John M Hicks" <>, 
> , 
> Subject: Re: multicast issue = 5828
> Reply-To: 
>
>
>
>
> On Thu, November 4, 2004 8:24 am, Marshall Eubanks said:
> > There are several possibilities. What OS / player are you using.
>
>   I'm using Quicktime in MAC OS X.  I also used Quicktime under Windows
> with the same result.  Both had worked fine to view this stream
> previously.
>
> >  Quicktime, for example, had a bug a good while ago that meant that it
> > would not
> > refresh IGMP member messages (or maybe it was the Mac stack, but that 
> > was
> > the effect).
> > So, groups will be joined but would then time out and die.
> >
> > What I was trying to make sure was that the problem was not in the
> > receiver computer -
> > thus the request to use rtpdump. (rtpqual uses the same code, so it 
> > means
> > that the problem is not in
> > the player).
>
>   Right.
>
> > The times to stop you are talking about are much too long to be due to
> > passing
> > encapsulated data but rejecting multicast, so it has to be either a 
> > data
> > stoppage or a local problem
> > (which includes the player, the player host, and your local DR).
> >
> > The next step would be to use tcpdump to see if IGMP membership 
> > messages
> > are being sent as they
> > should be.
>
>   I'm out of town today but I have checked the router and it shows the
> correct state indicating that my host is sending IGMP membership
> messages correctly.  It works fine for other streams.
>
>
> > Also, you could look in your DR AFTER the data stops. If the problem 
> > is
> > from the receiver, the group
> > will be missing. If it is due to a data stoppage, then the group state
> > will still be there, as
> > the DR will still be receiving IGMP member messages from the group.
>
>   Multicast state is still there and correct in the DR and in the 
> routers
> in the path towards the source, including the Abilene core routers.
>
>   A "show ip mroute x count" on my Cisco router that peers with the
> Northern Lights GigaPOP shows that traffc has stopped and is not
> reaching our network.
>
>   Looking at the Abilene core routers they show 0 pps and 0 Kbps but 
> have
> the correct multicast state, indicating that the traffic stops before
> reaching the Abilene backbone.
>
> > (Actually, quicktime sends RTCP traffic from each receiver, so that 
> > should
> > be visible going out
> > bound if you can use quicktime.)
> >
> > Thanks
> > Marshall
> >
> >
> > On Wed, 3 Nov 2004 22:32:42 -0600 (CST)
> >  "Bruce Curtis" 
> > <>
> >  wrote:
> > > On Wed, November 3, 2004 9:53 pm, Marshall Eubanks said:
> > > > On Wed, 3 Nov 2004 15:45:35 -0600
> > > >  Bruce Curtis 
> > > > <>
> > > >  wrote:
> > > > >
> > > >
> > > > Yesterday I got all of my Abilene routes back.
> > > >
> > > >
> > > > >    Today I'm receiving MBGP routes for 63.105.122.28 so the 
> > > > > multicast
> > > > > session starts, but now we are back to the other problem that I 
> > > > > only
> > > > > receive a few packets and then the session stops.
> > > >
> > > > Get a copy of rtpdump and issue
> > > >
> > > > rtpdump -F ascii  233.64.133.120/8022
> > > >
> > > > before you start the player up.
> > > >
> > > > You should see a flood of packets come through. If the
> > > > pkayer _still_ stops, then whether not rtpdump stops will tell
> > > > whether this a player issue or a network issue.
> > > >
> > > > Marshall
> > >
> > >   The abilene core routers show 0 Kbps and 0 pps after the initial 
> > > flow.
> > >
> > >
> > >   I used rtpqual, I got the output below fairly quickly and then 
> > > stopped
> > > receiving packets.  Earlier I also verified with tcpdump that it was 
> > > not
> > > a problem with the viewer, I could see that the packets stopped.  
> > > Also
> > > the viewer works fine with other streams.
> > >
> > > $ rtpqual 233.64.133.120 8022
> > > Defaulting to: rtpqual 233.64.133.120 8022 rtp
> > > Report from: rtpqual 233.64.133.120 8022 rtp at Wed Nov  3 22:25:36 
> > > 2004
> > > T Pkts Loss  % Late Bytes |  Pkts  Loss  %  Late     kB Sender
> > > 36  44   2   4   1  46366 |    44    2   4    1      45 63.105.122.28
> > > 37  29   2   6   0  29195 |    73    4   5    1      73 63.105.122.28
> > > 38  28   3   9   0  27462 |   101    7   6    1     100 63.105.122.28
> > > 39  30   2   6   0  31166 |   131    9   6    1     131 63.105.122.28
> > > 40  31   0   0   0  30988 |   162    9   5    1     161 63.105.122.28
> > >
> > >   As the time stamp shows there was a three minute pause and then I
> > > received another group of packets.
> > >
> > > 41   9   0   0   0   8650 |   171    9   5    1     169 63.105.122.28
> > > Report from: rtpqual 233.64.133.120 8022 rtp at Wed Nov  3 22:28:32 
> > > 2004
> > > T Pkts Loss  % Late Bytes |  Pkts  Loss  %  Late     kB Sender
> > > 31   4  12  75   0   4185 |   175   21  10    1     173 63.105.122.28
> > > 32  40   2   4   0  44018 |   215   23   9    1     216 63.105.122.28
> > > 33  34   0   0   0  40881 |   249   23   8    1     256 63.105.122.28
> > > 34  40   0   0   0  41966 |   289   23   7    1     297 63.105.122.28
> > > 35  40   0   0   0  44122 |   329   23   6    1     340 63.105.122.28
> > > 36  36   0   0   0  39468 |   365   23   5    1     379 63.105.122.28
> > > 37  33   0   0   0  37322 |   398   23   5    1     415 63.105.122.28
> > >
> > >   And then is stopped again, will likely have another burst in 3
> > > minutes...
> > >
> > >   Today I looked at the whole list of MBGP routes from level 3 and 
> > > found
> > > another site with an MSDP entry that came through level 3.  I used
> > > rtpqual for that site and it worked fine.  But it looked like that 
> > > site
> > > was in Chicago if I remember correctly.
> > >
> > >
> > >
> > >
> > > > > On Tuesday, November 2, 2004, at 01:25  PM, Bruce Curtis wrote:
> > > > >
> > > > > > On Tuesday, November 2, 2004, at 01:14  PM, John M Hicks wrote:
> > > > > >
> > > > > > > Bruce,
> > > > > > > Back at it again today.  Can you send me a snap shot of the
> > > multicast
> > > > > > > state
> > > > > > > for what you are looking at?  Also, where are you getting the SAs
> > > > > > > from?  Do
> > > > > > > you have a level3 peering?
> > > > > > > Thanks,
> > > > > > > -john
> > > > > >
> > > > > >   We don't have a level 3 peering, but the source 63.105.122.28 is
> > > on
> > > > > > level 3.  Our only multicast peering is with the Northern Lights
> > > > > > GigaPOP.
> > > > > >
> > > > > >   The problem is different that it was last week.  Now I get no
> > > > > > multicast packets from  63.105.122.28 for group 233.64.133.110,
> > > which
> > > > > > makes sense since I'm not receiving any routes for 63.105.122.28 
> > > > > > in
> > > > > > MBGP anymore, and so there is an rpf failure.  The email I sent
> > > > > > yesterday from that included the entries from the Level 3 looking
> > > > > > glass showed that there is a problem within the Level 3 network 
> > > > > > and
> > > > > > the MBGP announcements for 63.105.122.28 aren't making it to the
> > > West
> > > > > > Coast portion of Level 3's network.
> > > > > >
> > > > > >
> > > > > >
> > > > > > i2.ndsu>show ip mroute 233.64.133.110
> > > > > > IP Multicast Routing Table
> > > > > > Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C -
> > > > > > Connected,
> > > > > >        L - Local, P - Pruned, R - RP-bit set, F - Register flag,
> > > > > >        T - SPT-bit set, J - Join SPT, M - MSDP created entry,
> > > > > >        X - Proxy Join Timer Running, A - Candidate for MSDP
> > > > > > Advertisement,
> > > > > >        U - URD, I - Received Source Specific Host Report, Z -
> > > > > > Multicast Tunnel
> > > > > >        Y - Joined MDT-data group, y - Sending to MDT-data group
> > > > > > Outgoing interface flags: H - Hardware switched
> > > > > >  Timers: Uptime/Expires
> > > > > >  Interface state: Interface, Next-Hop or VCD, State/Mode
> > > > > >
> > > > > > (*, 233.64.133.110), 00:00:50/stopped, RP 134.129.65.254, flags: 
> > > > > > SP
> > > > > >   Incoming interface: Null, RPF nbr 0.0.0.0
> > > > > >   Outgoing interface list: Null
> > > > > >
> > > > > > (63.105.122.28, 233.64.133.110), 00:00:50/00:02:40, flags: P
> > > > > >   Incoming interface: FastEthernet1/0/0, RPF nbr 134.129.107.3
> > > > > >   Outgoing interface list: Null
> > > > > >
> > > > > >
> > > > > >
> > > > > > i2.ndsu>show ip mbgp 63.105.122.28
> > > > > > % Network not in table
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2.ndsu>show ip msdp sa 233.64.133.110
> > > > > > MSDP Source-Active Cache - 2 entries for 233.64.133.110
> > > > > > (63.105.122.28, 233.64.133.110), RP 206.61.163.252, MBGP/AS 1239,
> > > > > > 1w3d/00:05:13, Peer 192.42.152.174
> > > > > >
> > > > > >   Even though whois shows that 63.105.122.28 belongs to uunet and
> > > > > > 206.61.163.252 belongs to sprint traceroutes show that both are on
> > > > > > Level 3's network.
> > > > > >
> > > > > >
> > > > > > traceroute 206.61.163.252
> > > > > > traceroute to 206.61.163.252 (206.61.163.252), 30 hops max, 40 
> > > > > > byte
> > > > > > packets
> > > > > >  1  a095.not-a-bridge.ndsu.nodak.edu (134.129.95.100)  0.859 ms
> > > 0.779
> > > > > > ms  0.321 ms
> > > > > >  2  fast100.i1.ndsu.nodak.edu (134.129.107.3)  0.647 ms  0.444 ms
> > > > > > 0.576 ms
> > > > > >  3  router.gig.hecn.ndsu.nodak.edu (134.129.29.41)  1.166 ms  
> > > > > > 1.136
> > > ms
> > > > > >  0.739 ms
> > > > > >  4  165.234.165.131 (165.234.165.131)  1.634 ms  1.004 ms  1.287 
> > > > > > ms
> > > > > >  5  165.234.165.129 (165.234.165.129)  1.748 ms  1.677 ms  1.833 
> > > > > > ms
> > > > > >  6  sl-gw33-chi-0-0.sprintlink.net (144.223.34.197)  37.837 ms
> > > 38.19
> > > > > > ms  41.226 ms
> > > > > >  7  sl-bb22-chi-4-0.sprintlink.net (144.232.26.21)  38.921 ms
> > > 40.394
> > > > > > ms 206.61 38.611 ms
> > > > > >  8  sl-st21-chi-13-0.sprintlink.net (144.232.20.91)  39.264 ms
> > > 39.245
> > > > > > ms  41.346 ms
> > > > > >  9  sl-st20-chi-1-0.sprintlink.net (144.232.8.102)  39.151 ms
> > > 40.284
> > > > > > ms  39.616 ms
> > > > > > 10  so-2-1-0.edge1.chicago1.level3.net (209.0.225.21)  39.285 ms
> > > > > > 40.818 ms  38.925 ms
> > > > > > 11  so-2-1-0.bbr2.chicago1.level3.net (209.244.8.13)  51.131 ms
> > > > > > 88.688 ms  40.646 ms
> > > > > > 12  ge-0-3-0.bbr2.washington1.level3.net (64.159.0.229)  58.12 ms
> > > > > > so-2-0-0.bbr2.washington1.level3.net (209.247.10.130)  58.892 ms
> > > > > > 61.364 ms
> > > > > > 13  ge-7-1.ipcolo1.washington1.level3.net (4.68.121.75)  58.171 ms
> > > > > > ge-7-0.ipcolo1.washington1.level3.net (4.68.121.11)  58.313 ms
> > > > > > ge-9-1.ipcolo1.washington1.level3.net (4.68.121.107)  58.163 ms
> > > > > > 14  unknown.level3.net (63.210.25.154)  59.39 ms^C^\Quit
> > > > > >
> > > > > >
> > > > > >
> > > > > > ---
> > > > > > Bruce Curtis                            
> > > > > >
> > > > > > Certified NetAnalyst II                         701-231-8527
> > > > > > North Dakota State University
> > > > >
> > > > > ---
> > > > > Bruce Curtis                            
> > > > >
> > > > > Certified NetAnalyst II                         701-231-8527
> > > > > North Dakota State University
> > >
> > >
> > > ---
> > > Bruce Curtis                             
> > >
> > > Certified NetAnalyst II                                701-231-8527
> > > North Dakota State University
>
>
> ---
> Bruce Curtis                             
>
> Certified NetAnalyst II                                701-231-8527
> North Dakota State University
Matthew Davy
Chief Network Engineer, Indiana University
University Information Technology Services / Abilene Network Operations 
Center
2711 East 10th Street, Bloomington IN, 47403

 /  812.855.7728
PGP key fingerprint: A84D DFB6 9DD5 BEB4 1EF7  D713 956F F85C 6422 CBEB
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (Darwin)

iD8DBQFBipJKlW/4XGQiy+sRAl1vAKCpGrfWX0xjMAVC2qczuOb5XrZ42wCg0M13
W8VwCT0C2Q35av2Zg5mtA6U=
=6Gc+
-----END PGP SIGNATURE-----