All things related to multicast

["Marshall Eubanks" <>]

On Thu, 17 Jul 2003 20:36:25 +0300
 "Lehtonen Rami" 
<>
 wrote:
> This sounds a like a good time to have MCOP standardised and have a tool for
> operators to push filters to edge routers:) Or is the other solution to turn
> down the MSDP at some point?

Well, this storm is over (or is it - Level 3 lost Abilene multicast 
peering a little while
ago - is this connected?) BUT -
<full rant mode>
unless effort is put into fixing / replacing or hardening MSDP, this will kill
interdomain ASM. Having this happen will really hurt ALL multicast - who will
help prevent it ?
</full rant mode>

Regards
Marshall



> 
> - Rami
> 
> > -----Original Message-----
> > From: Marshall Eubanks 
> > [mailto:]
> > Sent: 17. heinäkuuta 2003 18:46
> > To: Alan Crosswell; JonAlf Dyrland-Weaver
> > Cc: 
> > ;
> >  wg-multicast; 
> > 
> > Subject: Re: another cisco log question
> > 
> > 
> > On Thu, 17 Jul 2003 11:15:36 -0400
> >  Alan Crosswell 
> > <>
> >  wrote:
> > > JonAlf Dyrland-Weaver wrote:
> > >  > ok, next time I'll wait till after I've gone through the 
> > entire log before
> > >  > I send an email. Same deal here, errors I've never seen 
> > before. These
> > >  > start at 9:47 and go on until the end of the log
> > >  >
> > 
> > Another SA storm is definitely underway - the worst since RAMEN
> > 
> > Thu Jul 17 06:12:42 2003 MSDP_Unique_Entries 14319 
> > MSDP_Unique_Groups 12792
> > MSDP_RPs 274 MSDP_Largest_Group_size 143 sources-groups 1527 
> > Thu Jul 17 00:12:41 2003 MSDP_Unique_Entries 14030 
> > MSDP_Unique_Groups 12547
> > MSDP_RPs 268 MSDP_Largest_Group_size 129 sources-groups 1483 
> > Sun Jul 13 06:12:41 2003 MSDP_Unique_Entries 12376 
> > MSDP_Unique_Groups 11111
> > MSDP_RPs 260 MSDP_Largest_Group_size 119 sources-groups 1265 
> > Sun Jun 22 00:12:40 2003 MSDP_Unique_Entries 11085 
> > MSDP_Unique_Groups 9565
> > MSDP_RPs 277 MSDP_Largest_Group_size 529 sources-groups 1520 
> > Sun Jun 22 06:12:41 2003 MSDP_Unique_Entries 9684 
> > MSDP_Unique_Groups 8194
> > MSDP_RPs 271 MSDP_Largest_Group_size 529 sources-groups 1490 
> > Thu May 29 18:12:41 2003 MSDP_Unique_Entries 9235 
> > MSDP_Unique_Groups 7695
> > MSDP_RPs 288 MSDP_Largest_Group_size 149 sources-groups 1540 
> > 
> > Their frequency and severity seems to be increasing :( - see 
> > 
> > http://www.multicasttech.com/status/msdp_sa_cache.plot.gif
> > (Figure 6a in
> > http://www.multicasttech.com/status/ )
> > 
> > Marshall
> > 
> > 
> > 
> > >  >        nn2k-gw:
> > >  >          21:47:37: SA from peer 199.109.5.6, RP 128.3.120.81 for
> > > (131.243.254.41, 
> > > 237.82.129.181) exceeded sa-limit of 10000
> > >  >          21:48:38: SA from peer 199.109.5.6, RP 128.3.120.81 for
> > > (131.243.254.41, 
> > > 227.68.78.94) exceeded sa-limit of 10000
> > >  >          21:49:47: SA from peer 199.109.5.6, RP 
> > 128.3.120.81 for
> > > (131.243.254.41, 
> > > 237.71.134.52) exceeded sa-limit of 10000
> > >  >
> > >  >          03:58:24: SA from peer 199.109.5.6, RP 128.3.120.81 for
> > > (131.243.254.41, 
> > > 235.186.250.82) exceeded sa-limit of 10000
> > >  >          03:59:24: SA from peer 199.109.5.6, RP 128.3.120.81 for
> > > (131.243.254.41, 
> > > 233.58.66.91) exceeded sa-limit of 10000
> > >  >
> > >  >
> > >  > -JonAlf
> > > 
> > > [I am CCing the I2 Multicast Working Group to see if I got 
> > any of this
> > > right.]
> > > 
> > > This is probably an attempted multicast DoS likely caused 
> > by a host infected 
> > > with Ramen or SQL Slammer.  They port-scan multicast IP 
> > addresses which
> > > generate 
> > > a Source Active (SA) for each multicast destination.  I'm 
> > not sure if the 
> > > threshold being exceeded is from the actual cause of the 
> > attack or just the
> > > one 
> > > that crosses the threshold of 10000.  I think unfortunately 
> > it's the harder 
> > > answer:  You need to do a show ip msdp count to see which 
> > AS has an unusually
> > > large number of SAs, then find that AS's contact with whois
> > > <as>@whois.arin.net. 
> > >   For example:
> > > 
> > > nnn2k-gw>sho ip msdp count
> > > SA State per Peer Counters, <Peer>: <# SA learned>
> > >      199.109.5.6: 2568
> > > 
> > > SA State per ASN Counters, <asn>: <# sources>/<# groups>
> > >      Total entries: 2568
> > >      3: 1/1, 8: 6/6, 9: 4/3, 17: 28/8
> > >      18: 12/4, 24: 1/1, 25: 34/11, 26: 7/3
> > >      32: 10/7, 38: 7/6, 47: 6/5, 55: 2/2
> > >      59: 24/14, 68: 1/1, 70: 1/1, 73: 18/9
> > >      81: 18/5, 87: 12/7, 88: 1/1, 102: 5/5
> > >      103: 7/6, 109: 178/148, 111: 2/1, 137: 6/6
> > >      145: 1/1, 159: 7/5, 160: 1/1, 194: 8/7
> > >      195: 4/1, 210: 6/3, 217: 2/2, 224: 33/30
> > >      225: 2/2, 237: 29/21, 261: 1/1, 271: 8/5
> > >      291: 1/1, 292: 11/4, 293: 116/38, 297: 6/6
> > >      377: 4/3, 549: 3/2, 553: 1/1, 559: 2/2
> > >      589: 3/2, 680: 54/33, 683: 24/12, 704: 41/27
> > >      766: 4/4, 776: 2/2, 777: 1/1, 1103: 31/22
> > >      1201: 5/2, 1206: 2/1, 1213: 3/3, 1224: 13/7
> > >      1239: 85/78, 1249: 1/1, 1653: 25/17, 1657: 4/4
> > >      1716: 8/6, 1739: 2/1, 1741: 8/7, 1742: 4/4
> > >      1781: 1/1, 1798: 2/2, 1835: 8/3, 1880: 3/1
> > >      1916: 6/5, 1936: 6/6, 1938: 1/1, 1998: 4/4
> > >      2055: 6/3, 2107: 2/2, 2193: 1/1, 2200: 68/34
> > >      2269: 1/1, 2381: 2/1, 2422: 9/7, 2496: 3/3
> > >      2546: 1/1, 2547: 1/1, 2594: 18/18, 2603: 1/1
> > >      2607: 43/43, 2611: 4/4, 2637: 3/3, 2698: 7/6
> > >      2701: 1/1, 2831: 43/20, 2833: 2/2, 2842: 2/2
> > >      2846: 6/3, 2852: 81/4, 3216: 12/12, 3303: 1/1
> > >      3323: 4/3, 3390: 2/2, 3450: 3/2, 3582: 188/184
> > >      3661: 16/16, 3676: 1/1, 3685: 80/4, 3807: 4/3
> > >      3912: 8/7, 3999: 12/7, 4130: 3/3, 4385: 1/1
> > >      4538: 20/7, 4621: 2/2, 4767: 15/15, 5050: 1/1
> > >      5408: 12/11, 5466: 1/1, 5640: 11/2, 5661: 37/28
> > >      5739: 9/7, 6192: 1/1, 6200: 8/3, 6263: 18/10
> > >      6342: 1/1, 6356: 19/8, 6360: 111/3, 6366: 43/3
> > >      6435: 2/2, 6509: 11/7, 6854: 5/5, 6867: 16/16
> > >      7018: 3/2, 7050: 5/5, 7212: 5/3, 7569: 1/1
> > >      7570: 2/2, 7572: 7/4, 7660: 6/2, 7896: 3/2
> > >      8071: 3/3, 8111: 2/1, 8617: 6/6, 9270: 9/9
> > >      9406: 10/8, 10326: 1/1, 10364: 3/3, 10421: 3/3
> > >      10437: 3/3, 10508: 8/2, 10546: 18/18, 10578: 1/1
> > >      10680: 6/6, 10702: 1/1, 10755: 7/7, 10876: 2/2
> > >      11039: 1/1, 11422: 1/1, 11537: 15/4, 11546: 15/12
> > >      11808: 3/2, 11809: 1/1, 12779: 27/1, 13501: 3/2
> > >      13778: 6/3, 14183: 2/2, 14348: 4/4, 16430: 9/4
> > >      16517: 17/17, 17579: 6/6, 18062: 3/2, 19149: 2/1
> > >      20130: 4/4, 20205: 1/1, 20894: 14/14, 20965: 293/125
> > >      22168: 8/6, 25631: 8/2, 25656: 1/1, 26046: 2/2
> > >      26406: 5/4, 64609: 4/2, 65026: 2/2, 65401: 25/15
> > > nn2k-gw>sho ip msdp count 20965
> > > n2k-gw>sho ip msdp count 20965
> > > SA State per ASN Counters, <asn>: <# sources>/<# groups>
> > >      Total entries: 2577
> > >      20965: 293/125
> > > 
> > > bash-2.05b$ whois 
> > > 
> > > 
> > > Of course, this information times out in 30 seconds, so 
> > looking at it at the 
> > > time you review the logs is not useful:-(
> > > 
> > > Perhaps someone can cobble toghether a tool to check this 
> > via SNMP or expect.
> > > I 
> > > think the Abilene NOC already does just this, so we may not need to.
> > > 
> > > We do need to check for our own multicast SAs jumping up 
> > which means we are 
> > > mDoSing like we did earlier this week.
> > > 
> > > I think this check involves looking at the PIM state on our outgoing
> > > multicast 
> > > interface:
> > > 
> > > nn2k-gw>sho ip mroute interface pos0/0/0
> > > IP Multicast Routing Table
> > > Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM 
> > Group, C - Connected,
> > >         L - Local, P - Pruned, R - RP-bit set, F - Register flag,
> > >         T - SPT-bit set, J - Join SPT, M - MSDP created entry,
> > >         X - Proxy Join Timer Running, A - Candidate for 
> > MSDP Advertisement,
> > >         U - URD, I - Received Source Specific Host Report, s - SSM
> > > Outgoing interface flags: H - Hardware switched
> > > Timers: Uptime/Expires
> > > Interface state: Interface, Next-Hop or VCD, State/Mode
> > > 
> > > (128.59.31.169, 224.2.211.27), 17:39:22/00:03:29, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 17:36:34/00:03:21
> > > 
> > > (*, 224.2.127.254), 7w0d/00:00:00, RP 128.59.0.15, flags: SJCL
> > >    Incoming interface: Null, RPF nbr 0.0.0.0
> > >    Outgoing interface list:
> > >      GigabitEthernet6/0/0, Forward/Sparse-Dense, 7w0d/00:03:29
> > >      POS0/0/0, Forward/Sparse, 7w0d/00:00:00
> > > 
> > > (128.59.31.169, 224.2.127.254), 00:00:14/00:03:15, flags: CLX
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:00:19/00:03:16
> > > 
> > > (128.59.31.187, 224.2.127.254), 2d15h/00:02:37, flags: CLTXA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 2d15h/00:03:16
> > > 
> > > (128.59.31.189, 224.2.127.254), 2d15h/00:03:13, flags: CLTXA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 2d15h/00:03:16
> > > 
> > > (128.59.244.235, 224.2.127.254), 7w0d/00:03:20, flags: CLTXA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.31
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 7w0d/00:03:16
> > > 
> > > (160.39.194.221, 224.0.1.76), 00:58:44/00:02:02, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.31
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:58:43/00:03:16
> > > 
> > > (160.39.246.193, 224.0.1.76), 16:53:24/00:02:01, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 16:52:39/00:03:15
> > > 
> > > (160.39.246.219, 224.0.1.76), 01:08:58/00:01:56, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 01:08:58/00:03:15
> > > 
> > > (160.39.247.20, 224.0.1.76), 21:00:57/00:02:16, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 21:00:27/00:03:15
> > > 
> > > (160.39.247.91, 224.0.1.76), 00:06:33/00:03:02, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:06:11/00:03:15
> > > 
> > > (160.39.247.92, 224.0.1.76), 00:28:42/00:01:53, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:27:59/00:03:15
> > > 
> > > (160.39.247.94, 224.0.1.76), 19:55:08/00:01:48, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 19:55:01/00:03:15
> > > 
> > > (160.39.247.95, 224.0.1.76), 00:28:54/00:01:54, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:28:54/00:03:15
> > > 
> > > (160.39.247.96, 224.0.1.76), 2d15h/00:03:10, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 2d15h/00:03:15
> > > 
> > > (160.39.247.97, 224.0.1.76), 1d00h/00:01:33, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 1d00h/00:03:15
> > > 
> > > (160.39.247.98, 224.0.1.76), 1d20h/00:01:40, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 1d20h/00:03:15
> > > 
> > > (160.39.247.100, 224.0.1.76), 02:01:21/00:02:57, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 02:00:36/00:03:15
> > > 
> > > (160.39.247.103, 224.0.1.76), 08:01:10/00:01:47, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 08:01:04/00:03:15
> > > 
> > > (160.39.247.104, 224.0.1.76), 1d23h/00:03:08, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 1d23h/00:03:15
> > > 
> > > (160.39.247.105, 224.0.1.76), 14:03:06/00:01:46, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 14:03:06/00:03:15
> > > 
> > > (160.39.247.106, 224.0.1.76), 08:01:10/00:01:46, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 08:01:04/00:03:14
> > > 
> > > (160.39.247.107, 224.0.1.76), 00:06:33/00:03:02, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:06:11/00:03:14
> > > 
> > > (160.39.247.108, 224.0.1.76), 1d16h/00:03:08, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 1d16h/00:03:14
> > > 
> > > (160.39.247.109, 224.0.1.76), 20:15:49/00:01:50, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 20:15:49/00:03:14
> > > 
> > > (160.39.247.111, 224.0.1.76), 19:49:32/00:02:38, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 19:48:56/00:03:14
> > > 
> > > (160.39.247.112, 224.0.1.76), 00:23:18/00:02:47, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:23:18/00:03:14
> > > 
> > > (160.39.247.113, 224.0.1.76), 00:56:36/00:01:57, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:56:36/00:03:14
> > > 
> > > (160.39.247.207, 224.0.1.76), 00:06:28/00:03:06, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:06:12/00:03:14
> > > 
> > > (128.59.153.50, 224.0.1.85), 00:03:14/00:00:15, flags: X
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.21
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:03:14/00:03:14
> > > 
> > > (128.59.31.156, 224.0.1.84), 2d01h/00:03:23, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 2d01h/00:03:14
> > > 
> > > (128.59.31.169, 224.2.241.169), 17:36:31/00:03:29, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 00:47:29/00:03:14
> > > 
> > > (128.59.31.187, 224.2.133.134), 2d15h/00:01:55, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 2d15h/00:03:14
> > > 
> > > (128.59.31.189, 224.2.133.134), 2d15h/00:01:35, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 2d15h/00:03:14
> > > 
> > > (128.59.244.235, 224.2.133.134), 2d15h/00:02:13, flags: TA
> > >    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.31
> > >    Outgoing interface list:
> > >      POS0/0/0, Forward/Sparse, 2d15h/00:03:14
> > > 
> > > 
> > > And see if there's an unusual number or unusual sources.... 
> > Hmmm.. look at
> > > that, 
> > > we are leaking a bunch of stuff including IAPP.MCAST.NET 
> > which appears to be 
> > > related to Lucent Wireless AP's: 
> > > 
> http://archives.internet2.edu/guest/archives/wg-multicast/log200303/msg00003.html
> > 
> > I guess it's time to update our SA filter list.  Looks like we may be able
> to
> > find rogue AP's this way too:-)
> > 
> > Ditto for JINI-REQUEST?  Looks like they are coming from our Ninja 
> > printing
> > stations.
> > 
> > /a
> > 
> > 
>