All things related to multicast

[Havard Eidnes <>]

Hi,

we're running two beacon servers inside UNINETT, and this evening I
decided I'd take a closer look at why our multicast connectivity has
been deteriorating and currently is as bad as it is (I'm the
"vever.runit.no" entry in the table).

I've come across a problem that has me stumped, and I was hoping
someone else could easily spot what the cause is.  I'm not able to
see whether this is a Cisco bug, or a misconfiguration of our
PIM-SM/MSDP/MBGP setups.

More or less at random I picked one of the sources many others are
able to receive, qlog.nerdc.ufl.edu at 128.227.128.34 (subsequent
probing reveals that this particular problem seems to affect many
other sources on the beacon list as well).

An mtrace from my beacon towards that source (via the beacon group)
shows among others these three entries:

 -5  oslo-gw1.uninett.no (128.39.0.249)  PIM/MBGP  thresh^ 0  RP/Core
 -6  s-gw.nordu.net (193.10.252.229)  PIM  thresh^ 1  Prune sent upstream
 -7  sw-gw.nordu.net (193.10.252.211)  PIM/MBGP  thresh^ 0  RP/Core

(Complete mtrace output attached below; let it slide for now that it
trails off somewhere beyond Abilene and doesn't reach the source...)

I'm able to tweak the config on all those routers, and I'm thus also
in some way responsible for the mess... ;-)  Looking at the multicast
routing table on s-gw, I find:

(128.227.128.34, 233.2.171.1), 00:02:27/00:01:02, flags:
  Incoming interface: SRP1/0, RPF nbr 193.10.252.211, Mbgp
  Outgoing interface list:
    POS0/1, Forward/Sparse, 00:02:27/00:03:22 (ttl-threshold 1)

i.e. no trace of a "pruned" flag.  However, repeated mtraces still
give me the "Prune sent upstream", and no traffic forwarded through
s-gw.  Yes, I've verified that sw-gw is indeed (claiming to) forward
traffic towards s-gw, its entry says:

(128.227.128.34, 233.2.171.1), 3d11h/00:03:22, flags: T
  Incoming interface: POS0/1, RPF nbr 193.10.252.138, Mbgp
  Outgoing interface list:
    SRP4/0, Forward/Sparse, 2d21h/00:02:25

and the corresponding forwarding count is steadily increasing; these
are a few seconds apart:

Group: 233.2.171.1, Source count: 41, Group pkt count: 22890735
  Source: 128.227.128.34/32, Forwarding: 2230002/9/103/6, Other: 6/0/0
...
Group: 233.2.171.1, Source count: 22, Group pkt count: 22891652
  Source: 128.227.128.34/32, Forwarding: 2230093/9/103/6, Other: 6/0/0

Clearing the multicast routing table on sw-gw and s-gw also appear
to do nothing to resolve the problem.

The MSDP peering session between UNINETT and NORDUnet is between
oslo-gw1 and sw-gw (which currently has MSDP caching turned off),
i.e. s-gw doesn't speek MSDP and is not an RP; external MBGP is
spoken between oslo-gw1 and s-gw.

The software versions involved here are

   sw-gw    (GSR-P-M), Version 12.0(13.6)S2
   s-gw     (GSR-P-M), Version 12.0(10.6)S2
   oslo-gw1 (GSR-P-M), Version 12.0(14.6)S1

i.e. not the absolutely latest and greatest, but relatively recent
images.

Anyone have a clue to share or a "redirect" for me?


Best regards,

- Håvard
vever# mtrace qlog.nerdc.ufl.edu vever 233.2.171.1
Mtrace from 128.227.128.34 to 129.241.100.111 via group 233.2.171.1
Querying full reverse path... * switching to hop-by-hop:
  0  vever.runit.no (129.241.100.111)
 -1  runit-gw.runit.no (129.241.100.1)  PIM  thresh^ 0  
 -2  sg-gw.runit.no (129.241.249.7)  PIM  thresh^ 0  
 -3  trdS-gw.uninett.no (129.241.249.19)  PIM  thresh^ 0  
 -4  trd-gw.uninett.no (158.38.0.226)  PIM  thresh^ 0  
 -5  oslo-gw1.uninett.no (128.39.0.249)  PIM/MBGP  thresh^ 0  RP/Core
 -6  s-gw.nordu.net (193.10.252.229)  PIM  thresh^ 1  Prune sent upstream
 -7  sw-gw.nordu.net (193.10.252.211)  PIM/MBGP  thresh^ 0  RP/Core
 -8  us-gw.nordu.net (193.10.252.138)  PIM/MBGP  thresh^ 0  RP/Core
 -9  nycm-nordunet.abilene.ucaid.edu (193.10.252.74)  PIM/MBGP  thresh^ 0  
RP/Core
-10  wash-nycm.abilene.ucaid.edu (198.32.8.45)  PIM/MBGP  thresh^ 0  RP/Core
-11  atla-wash.abilene.ucaid.edu (198.32.8.65)  PIM/MBGP  thresh^ 0  RP/Core
-12  a713.c12008.atla.abilene.ucaid.edu (192.80.53.46)  PIM  thresh^ 0  
RP/Core
-13  * * * * ssrb-core-msfc-S3P7.ns.ufl.edu (128.227.252.153) didn't respond
-14  * * * 
-15  * * * 
-16  * * * ...giving up
Round trip time 178 ms


Results after 39 seconds:

  Source        Response Dest    Packet Statistics For     Only For Traffic
   * * *        129.241.100.111  All Multicast Traffic     From 128.227.128.34
     v       __/  rtt  149 ms    Lost/Sent = Pct  Rate       To 233.2.171.1
128.227.252.154
192.80.53.46    a713.c12008.atla.abilene.ucaid.edu RP/Core
     v     ^      ttl    0        4/364  =  1%   9 pps    10/364  =  3%   9 
pps
192.80.53.45   
198.32.8.65     atla-wash.abilene.ucaid.edu RP/Core
     v     ^      ttl    1     -128/7866 = -1% 201 pps   -10/354  = -2%   9 
pps
198.32.8.66    
198.32.8.45     wash-nycm.abilene.ucaid.edu RP/Core
     v     ^      ttl    2       90/7977 =  1% 204 pps    -1/364  =  0%   9 
pps
198.32.8.46    
193.10.252.74   nycm-nordunet.abilene.ucaid.edu RP/Core
     v     ^      ttl    3      479/19146=  3% 490 pps     9/365  =  2%   9 
pps
193.10.252.73  
193.10.252.138  us-gw.nordu.net RP/Core
     v     ^      ttl    4    -1006/20750= -4% 532 pps    -8/356  = -1%   9 
pps
193.10.252.137 
193.10.252.211  sw-gw.nordu.net RP/Core
     v     ^      ttl    5    20153/27523= 73% 705 pps   364/364  =100%   9 
pps
193.10.252.210 
193.10.252.229  s-gw.nordu.net Prune sent upstream
     v     ^      ttl    6        0/1447 =  0%  37 pps     0/0    = --%   0 
pps
193.10.252.230 
128.39.0.249    oslo-gw1.uninett.no RP/Core
     v     ^      ttl    7      -32/7604 =  0% 194 pps     0/0    = --%   0 
pps
128.39.0.250   
158.38.0.226    trd-gw.uninett.no 
     v     ^      ttl    8    -5103/1510 = --%  38 pps     0/0    = --%   0 
pps
158.38.0.227   
129.241.249.19  trdS-gw.uninett.no 
     v     ^      ttl    9        0/1775 =  0%  45 pps     0/0    = --%   0 
pps
129.241.249.17 
129.241.249.7   sg-gw.runit.no 
     v     ^      ttl   10        0/1775 =  0%  45 pps     0/0    = --%   0 
pps
129.241.249.6  
129.241.100.1   runit-gw.runit.no 
     v      \__   ttl   11          1775        45 pps       0            0 
pps
129.241.100.111 129.241.100.111
  Receiver      Query Source