All things related to multicast

["Marshall Eubanks" <>]

On Thu, 17 Jul 2003 11:15:36 -0400
 Alan Crosswell 
<>
 wrote:
> JonAlf Dyrland-Weaver wrote:
>  > ok, next time I'll wait till after I've gone through the entire log 
> before
>  > I send an email. Same deal here, errors I've never seen before. These
>  > start at 9:47 and go on until the end of the log
>  >

Another SA storm is definitely underway - the worst since RAMEN

Thu Jul 17 06:12:42 2003 MSDP_Unique_Entries 14319 MSDP_Unique_Groups 12792
MSDP_RPs 274 MSDP_Largest_Group_size 143 sources-groups 1527 
Thu Jul 17 00:12:41 2003 MSDP_Unique_Entries 14030 MSDP_Unique_Groups 12547
MSDP_RPs 268 MSDP_Largest_Group_size 129 sources-groups 1483 
Sun Jul 13 06:12:41 2003 MSDP_Unique_Entries 12376 MSDP_Unique_Groups 11111
MSDP_RPs 260 MSDP_Largest_Group_size 119 sources-groups 1265 
Sun Jun 22 00:12:40 2003 MSDP_Unique_Entries 11085 MSDP_Unique_Groups 9565
MSDP_RPs 277 MSDP_Largest_Group_size 529 sources-groups 1520 
Sun Jun 22 06:12:41 2003 MSDP_Unique_Entries 9684 MSDP_Unique_Groups 8194
MSDP_RPs 271 MSDP_Largest_Group_size 529 sources-groups 1490 
Thu May 29 18:12:41 2003 MSDP_Unique_Entries 9235 MSDP_Unique_Groups 7695
MSDP_RPs 288 MSDP_Largest_Group_size 149 sources-groups 1540 

Their frequency and severity seems to be increasing :( - see 

http://www.multicasttech.com/status/msdp_sa_cache.plot.gif
(Figure 6a in
http://www.multicasttech.com/status/ )

Marshall



>  >    nn2k-gw:
>  >      21:47:37: SA from peer 199.109.5.6, RP 128.3.120.81 for
> (131.243.254.41, 
> 237.82.129.181) exceeded sa-limit of 10000
>  >      21:48:38: SA from peer 199.109.5.6, RP 128.3.120.81 for
> (131.243.254.41, 
> 227.68.78.94) exceeded sa-limit of 10000
>  >      21:49:47: SA from peer 199.109.5.6, RP 128.3.120.81 for
> (131.243.254.41, 
> 237.71.134.52) exceeded sa-limit of 10000
>  >
>  >      03:58:24: SA from peer 199.109.5.6, RP 128.3.120.81 for
> (131.243.254.41, 
> 235.186.250.82) exceeded sa-limit of 10000
>  >      03:59:24: SA from peer 199.109.5.6, RP 128.3.120.81 for
> (131.243.254.41, 
> 233.58.66.91) exceeded sa-limit of 10000
>  >
>  >
>  > -JonAlf
> 
> [I am CCing the I2 Multicast Working Group to see if I got any of this
> right.]
> 
> This is probably an attempted multicast DoS likely caused by a host 
> infected 
> with Ramen or SQL Slammer.  They port-scan multicast IP addresses which
> generate 
> a Source Active (SA) for each multicast destination.  I'm not sure if the 
> threshold being exceeded is from the actual cause of the attack or just the
> one 
> that crosses the threshold of 10000.  I think unfortunately it's the harder 
> answer:  You need to do a show ip msdp count to see which AS has an 
> unusually
> large number of SAs, then find that AS's contact with whois
> <as>@whois.arin.net. 
>   For example:
> 
> nnn2k-gw>sho ip msdp count
> SA State per Peer Counters, <Peer>: <# SA learned>
>      199.109.5.6: 2568
> 
> SA State per ASN Counters, <asn>: <# sources>/<# groups>
>      Total entries: 2568
>      3: 1/1, 8: 6/6, 9: 4/3, 17: 28/8
>      18: 12/4, 24: 1/1, 25: 34/11, 26: 7/3
>      32: 10/7, 38: 7/6, 47: 6/5, 55: 2/2
>      59: 24/14, 68: 1/1, 70: 1/1, 73: 18/9
>      81: 18/5, 87: 12/7, 88: 1/1, 102: 5/5
>      103: 7/6, 109: 178/148, 111: 2/1, 137: 6/6
>      145: 1/1, 159: 7/5, 160: 1/1, 194: 8/7
>      195: 4/1, 210: 6/3, 217: 2/2, 224: 33/30
>      225: 2/2, 237: 29/21, 261: 1/1, 271: 8/5
>      291: 1/1, 292: 11/4, 293: 116/38, 297: 6/6
>      377: 4/3, 549: 3/2, 553: 1/1, 559: 2/2
>      589: 3/2, 680: 54/33, 683: 24/12, 704: 41/27
>      766: 4/4, 776: 2/2, 777: 1/1, 1103: 31/22
>      1201: 5/2, 1206: 2/1, 1213: 3/3, 1224: 13/7
>      1239: 85/78, 1249: 1/1, 1653: 25/17, 1657: 4/4
>      1716: 8/6, 1739: 2/1, 1741: 8/7, 1742: 4/4
>      1781: 1/1, 1798: 2/2, 1835: 8/3, 1880: 3/1
>      1916: 6/5, 1936: 6/6, 1938: 1/1, 1998: 4/4
>      2055: 6/3, 2107: 2/2, 2193: 1/1, 2200: 68/34
>      2269: 1/1, 2381: 2/1, 2422: 9/7, 2496: 3/3
>      2546: 1/1, 2547: 1/1, 2594: 18/18, 2603: 1/1
>      2607: 43/43, 2611: 4/4, 2637: 3/3, 2698: 7/6
>      2701: 1/1, 2831: 43/20, 2833: 2/2, 2842: 2/2
>      2846: 6/3, 2852: 81/4, 3216: 12/12, 3303: 1/1
>      3323: 4/3, 3390: 2/2, 3450: 3/2, 3582: 188/184
>      3661: 16/16, 3676: 1/1, 3685: 80/4, 3807: 4/3
>      3912: 8/7, 3999: 12/7, 4130: 3/3, 4385: 1/1
>      4538: 20/7, 4621: 2/2, 4767: 15/15, 5050: 1/1
>      5408: 12/11, 5466: 1/1, 5640: 11/2, 5661: 37/28
>      5739: 9/7, 6192: 1/1, 6200: 8/3, 6263: 18/10
>      6342: 1/1, 6356: 19/8, 6360: 111/3, 6366: 43/3
>      6435: 2/2, 6509: 11/7, 6854: 5/5, 6867: 16/16
>      7018: 3/2, 7050: 5/5, 7212: 5/3, 7569: 1/1
>      7570: 2/2, 7572: 7/4, 7660: 6/2, 7896: 3/2
>      8071: 3/3, 8111: 2/1, 8617: 6/6, 9270: 9/9
>      9406: 10/8, 10326: 1/1, 10364: 3/3, 10421: 3/3
>      10437: 3/3, 10508: 8/2, 10546: 18/18, 10578: 1/1
>      10680: 6/6, 10702: 1/1, 10755: 7/7, 10876: 2/2
>      11039: 1/1, 11422: 1/1, 11537: 15/4, 11546: 15/12
>      11808: 3/2, 11809: 1/1, 12779: 27/1, 13501: 3/2
>      13778: 6/3, 14183: 2/2, 14348: 4/4, 16430: 9/4
>      16517: 17/17, 17579: 6/6, 18062: 3/2, 19149: 2/1
>      20130: 4/4, 20205: 1/1, 20894: 14/14, 20965: 293/125
>      22168: 8/6, 25631: 8/2, 25656: 1/1, 26046: 2/2
>      26406: 5/4, 64609: 4/2, 65026: 2/2, 65401: 25/15
> nn2k-gw>sho ip msdp count 20965
> n2k-gw>sho ip msdp count 20965
> SA State per ASN Counters, <asn>: <# sources>/<# groups>
>      Total entries: 2577
>      20965: 293/125
> 
> bash-2.05b$ whois 
> 
> 
> Of course, this information times out in 30 seconds, so looking at it at 
> the 
> time you review the logs is not useful:-(
> 
> Perhaps someone can cobble toghether a tool to check this via SNMP or 
> expect.
> I 
> think the Abilene NOC already does just this, so we may not need to.
> 
> We do need to check for our own multicast SAs jumping up which means we are 
> mDoSing like we did earlier this week.
> 
> I think this check involves looking at the PIM state on our outgoing
> multicast 
> interface:
> 
> nn2k-gw>sho ip mroute interface pos0/0/0
> IP Multicast Routing Table
> Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
>         L - Local, P - Pruned, R - RP-bit set, F - Register flag,
>         T - SPT-bit set, J - Join SPT, M - MSDP created entry,
>         X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
>         U - URD, I - Received Source Specific Host Report, s - SSM
> Outgoing interface flags: H - Hardware switched
> Timers: Uptime/Expires
> Interface state: Interface, Next-Hop or VCD, State/Mode
> 
> (128.59.31.169, 224.2.211.27), 17:39:22/00:03:29, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 17:36:34/00:03:21
> 
> (*, 224.2.127.254), 7w0d/00:00:00, RP 128.59.0.15, flags: SJCL
>    Incoming interface: Null, RPF nbr 0.0.0.0
>    Outgoing interface list:
>      GigabitEthernet6/0/0, Forward/Sparse-Dense, 7w0d/00:03:29
>      POS0/0/0, Forward/Sparse, 7w0d/00:00:00
> 
> (128.59.31.169, 224.2.127.254), 00:00:14/00:03:15, flags: CLX
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:00:19/00:03:16
> 
> (128.59.31.187, 224.2.127.254), 2d15h/00:02:37, flags: CLTXA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 2d15h/00:03:16
> 
> (128.59.31.189, 224.2.127.254), 2d15h/00:03:13, flags: CLTXA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 2d15h/00:03:16
> 
> (128.59.244.235, 224.2.127.254), 7w0d/00:03:20, flags: CLTXA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.31
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 7w0d/00:03:16
> 
> (160.39.194.221, 224.0.1.76), 00:58:44/00:02:02, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.31
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:58:43/00:03:16
> 
> (160.39.246.193, 224.0.1.76), 16:53:24/00:02:01, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 16:52:39/00:03:15
> 
> (160.39.246.219, 224.0.1.76), 01:08:58/00:01:56, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 01:08:58/00:03:15
> 
> (160.39.247.20, 224.0.1.76), 21:00:57/00:02:16, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 21:00:27/00:03:15
> 
> (160.39.247.91, 224.0.1.76), 00:06:33/00:03:02, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:06:11/00:03:15
> 
> (160.39.247.92, 224.0.1.76), 00:28:42/00:01:53, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:27:59/00:03:15
> 
> (160.39.247.94, 224.0.1.76), 19:55:08/00:01:48, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 19:55:01/00:03:15
> 
> (160.39.247.95, 224.0.1.76), 00:28:54/00:01:54, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:28:54/00:03:15
> 
> (160.39.247.96, 224.0.1.76), 2d15h/00:03:10, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 2d15h/00:03:15
> 
> (160.39.247.97, 224.0.1.76), 1d00h/00:01:33, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 1d00h/00:03:15
> 
> (160.39.247.98, 224.0.1.76), 1d20h/00:01:40, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 1d20h/00:03:15
> 
> (160.39.247.100, 224.0.1.76), 02:01:21/00:02:57, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 02:00:36/00:03:15
> 
> (160.39.247.103, 224.0.1.76), 08:01:10/00:01:47, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 08:01:04/00:03:15
> 
> (160.39.247.104, 224.0.1.76), 1d23h/00:03:08, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 1d23h/00:03:15
> 
> (160.39.247.105, 224.0.1.76), 14:03:06/00:01:46, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 14:03:06/00:03:15
> 
> (160.39.247.106, 224.0.1.76), 08:01:10/00:01:46, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 08:01:04/00:03:14
> 
> (160.39.247.107, 224.0.1.76), 00:06:33/00:03:02, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:06:11/00:03:14
> 
> (160.39.247.108, 224.0.1.76), 1d16h/00:03:08, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 1d16h/00:03:14
> 
> (160.39.247.109, 224.0.1.76), 20:15:49/00:01:50, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 20:15:49/00:03:14
> 
> (160.39.247.111, 224.0.1.76), 19:49:32/00:02:38, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 19:48:56/00:03:14
> 
> (160.39.247.112, 224.0.1.76), 00:23:18/00:02:47, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:23:18/00:03:14
> 
> (160.39.247.113, 224.0.1.76), 00:56:36/00:01:57, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:56:36/00:03:14
> 
> (160.39.247.207, 224.0.1.76), 00:06:28/00:03:06, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.41
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:06:12/00:03:14
> 
> (128.59.153.50, 224.0.1.85), 00:03:14/00:00:15, flags: X
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.21
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:03:14/00:03:14
> 
> (128.59.31.156, 224.0.1.84), 2d01h/00:03:23, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 2d01h/00:03:14
> 
> (128.59.31.169, 224.2.241.169), 17:36:31/00:03:29, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 00:47:29/00:03:14
> 
> (128.59.31.187, 224.2.133.134), 2d15h/00:01:55, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 2d15h/00:03:14
> 
> (128.59.31.189, 224.2.133.134), 2d15h/00:01:35, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.51
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 2d15h/00:03:14
> 
> (128.59.244.235, 224.2.133.134), 2d15h/00:02:13, flags: TA
>    Incoming interface: GigabitEthernet6/0/0, RPF nbr 128.59.1.31
>    Outgoing interface list:
>      POS0/0/0, Forward/Sparse, 2d15h/00:03:14
> 
> 
> And see if there's an unusual number or unusual sources.... Hmmm.. look at
> that, 
> we are leaking a bunch of stuff including IAPP.MCAST.NET which appears to 
> be 
> related to Lucent Wireless AP's: 
> http://archives.internet2.edu/guest/archives/wg-multicast/log200303/msg00003.html
> 
> I guess it's time to update our SA filter list.  Looks like we may be able 
> to
> find rogue AP's this way too:-)
> 
> Ditto for JINI-REQUEST?  Looks like they are coming from our Ninja printing 
> stations.
> 
> /a
> 
>