Internet2 Network Security SIG

[David Farmer <>]

Jeff,

Thanks for the response.  I agree we need to document and teach to our community R&E networking BGP Best Practices, these would be similar to the Best Practices for ISPs but with the added wrinkle that in our community we Local Perf R&E routes over commercial Internet routes, because of this we need to take additional precaution, to not hurt each other.

Also, I agree that Internet2 filtering hides routing policy issues that connectors and international partners may have.  Thinking about this, I'd like to suggest Internet2 provide reporting on what is being accepted, what is being filtered, and why to connectors and international partners. Maybe through a portal, with maybe the ability to have the portal provide a monthly snapshot emailed to the appropriate support contact.  As an example of what I mean take a look a http://routing.he.net/?cmd=search&pattern=57 or the Google ISP portal.

Also, the NOC at Chris Robb's prompting is taking a look at the routes I pointed out.

Thanks.

On Mon, Dec 30, 2019 at 11:12 AM Jeff Bartig <> wrote:

Dave,

There are multiple issues happening here, as you've noted.

First, there are routes being leaked from various networks/campuses that probably shouldn't be leaked.  We talked about this type of issue on the last Routing Security Office Hours call we held.  Networks need to use more than a simple prefix list in order to select their customer routes to advertise to peers and upstream providers.  BGP community tags are a very useful tool to help solve this problem.  If you have downstream BGP customers, then on ingress, these routes can be tagged with an informational BGP community that identifies that these routes were learned from a customer.  When the routes are exported to a peer or upstream provider, then you can select the routes based on them having the customer BGP community tag.

If you simply use a prefix list, then it opens up the possibility that you will eventually take a matching route learned from a peer or paid transit provider and advertise that route to peers and upstream providers.  I'm guessing that is what happened in many of these cases.  These types of problems can remain hidden, as long as you have your real customer routes with a higher local preference.  The leaks only happen when the higher preference customer route is no longer in the network's routing table.  This could be a temporary situation due to an outage or a permanent situation due to the downstream network no longer being a customer.  In those situations, the best path could be via paid transit or other peering, rather than a direct customer neighbor.  A simple prefix list used on upstream and peer neighbors will allow this route to leak.

The second issue is that networks propagated these routes.  While some of these routes were learned by Internet2 from peers, many came from Connectors/Participants.  For every customer BGP neighbor, Internet2 does have a prefix list that only accepts prefixes that belong to the downstream participants.  Internet2 also tags these routes with a customer BGP community tag.  Internet2 does have a sanity AS path filter to drop routes with commercial transit ASNs in the path.  Looking through the router configs, there are several of these ASN lists that are used depending upon the type of neighbor (customer or peer), v4 vs v6, and which Internet2 network (R&E or I2PX).  Some of these lists are incomplete and way out of date.  When they get updated, they should block most of what you've identified below.

Many of the leaks you identified happened more than one AS hop away from Internet2.  While Internet2 cleaning up its sanity AS path filtering will help, the networks between Internet2 and the leak should be implementing similar AS path filtering.  If they are allowing these leaked routes to be advertised to Internet2, then they are possibly also advertising the leaked routes to other peers and upstream transit providers.  Although Internet2's filtering is incomplete, it is still successfully blocking routes with many commercial ASNs in the path.  This is probably hiding the scope of this issue at the regional network level.  Had Internet2's AS path filters been up to date, this would completely hide visibility of this issue from an Internet2 perspective.

The topic of reintroducing BGP routing tutorials has come up many times recently.  I think the issues you've noticed are another indicator that there is a need for tutorials and better information for our community on BGP best practices.

Jeff



David Farmer wrote on 12/28/19 12:34 PM:

I understand it a game of whack-a-mole, but them moles need whacking, at least once a while or you'll get overrun by the moles. 😀

Furthermore, I think the is a corollary to the teach a man to fish parable. Probably something like, if teach a man to whack his own moles and you will have fewer moles to whack.  😀

But seriously, enough philosophy for a Saturday morning. I think we need to push the idea of a routing policy based on filtering and tagging on ingress from customers and exporting only routes that are tagged as coming from customers, instead a policy based on filtering on egress than many seem to be doing.

Thanks.

On Sat, Dec 28, 2019 at 12:04 PM Chris Robb <> wrote:

Forwarding to the noc to get this cleaned up. We do have sanity filters the block common commercial AS numbers but it’s probably been a while since the list has been updated. Filtering out R&E peer routes is harder and we definitely see some networks that are less disciplined with their route advertisements that pop up every now and then. It’s a bit of a game of whack-a-mole unfortunately. 

Sent from my iPhone

On Dec 28, 2019, at 12:51 PM, David Farmer <> wrote:



I'm sorry for cross-posting and for naming and shaming, but I think this needs some attention.

These I2 R&E routes all have major commercial transit providers in their AS Paths, a couple even more than one, and one is recirculating an ESNet route via a comercial ISP.

*> 42.83.130.0/24     146.57.255.241        2735    202      0 11537 22388 7660 7497 4635 6939 24785 8763 8763 8763 8763 24151 i
*> 42.83.132.0/24     146.57.255.241        2735    202      0 11537 22388 7660 7497 4635 6939 24785 8763 8763 8763 8763 24151 i
*> 42.83.137.0/24     146.57.255.241        2735    202      0 11537 22388 7660 7497 4635 6939 24785 8763 8763 8763 8763 24151 i

*> 82.194.8.0/24      146.57.255.241        2188    202      0 11537 20965 202993 3356 29049 29584 i

*> 98.179.129.0/24    146.57.255.241        3358    202      0 11537 62600 209 209 209 209 12189 20454 53372 i

*> 103.26.196.0/24    146.57.255.241        3809    202      0 11537 23855 23855 24514 3257 132354 132874 i
*> 119.40.112.0/24    146.57.255.241        3809    202      0 11537 23855 23855 24514 3257 9930 38868 38868 38868 38868 ?
*> 119.40.124.0/24    146.57.255.241        3809    202      0 11537 23855 23855 24514 3257 9930 38868 38868 38868 38868 ?

*> 125.208.34.0/24    146.57.255.241        2735    202      0 11537 22388 7660 7497 4635 6939 24785 8763 8763 8763 8763 24151 i
*> 125.208.41.0/24    146.57.255.241        2735    202      0 11537 22388 7660 7497 4635 6939 24785 8763 8763 8763 8763 24151 i

*> 170.158.66.0/23    146.57.255.241        1379    202      0 11537 3754 46158 46158 46158 46158 46158 46887 3356 6453 55002 i
*  192.188.178.0/23   146.57.255.241        2566    202      0 11537 10466 88 6939 293 293 293 50 i

*> 195.162.72.0/23    146.57.255.241        2751    202      0 11537 8895 3356 57344 197304 i

*> 195.189.212.0/24   146.57.255.241        2751    202      0 11537 8895 6453 6762 39386 25233 i

*> 199.59.212.0/22    146.57.255.241        2693    202      0 11537 81 3356 19271 29901 i

*> 199.125.69.0/24    146.57.255.241        2751    202      0 11537 10578 14325 6939 7922 14265 i

*> 202.45.133.0/24    146.57.255.241        3809    202      0 11537 23855 23855 24514 3257 45630 24314 i

*> 203.119.28.0/24    146.57.255.241        2735    202      0 11537 22388 7660 7497 4635 6939 24785 8763 8763 8763 8763 24151 i

*  203.119.33.0/24    146.57.255.241        2735    202      0 11537 22388 7660 7497 3491 21859 24151 i

Probably even worse these have major commercial transit providers and I2PX in their AS Paths 

*> 24.199.205.0/24    146.57.255.241        2693    202      0 11537 81 11164 7843 11426 i
*> 64.5.147.0/24      146.57.255.241        2143    202      0 11537 40220 11164 22773 i
*> 65.254.166.0/24    146.57.255.241        2143    202      0 11537 40220 11164 6939 22299 i
*> 65.254.181.0/24    146.57.255.241        2143    202      0 11537 40220 11164 6939 22299 i
*> 65.254.182.0/24    146.57.255.241        2143    202      0 11537 40220 11164 6939 22299 i
*> 65.254.183.0/24    146.57.255.241        2143    202      0 11537 40220 11164 6939 22299 i
*> 65.254.184.0/24    146.57.255.241        2143    202      0 11537 40220 11164 6939 22299 47036 i
*> 65.254.185.0/24    146.57.255.241        2143    202      0 11537 40220 11164 6939 22299 47036 i
*> 128.82.0.0/16      146.57.255.241        2143    202      0 11537 40220 11164 22773 1201 1201 1201 1201 ?
*> 137.198.0.0/16     146.57.255.241        2143    202      0 11537 40220 11164 22773 14655 i
*> 151.188.0.0/16     146.57.255.241        2143    202      0 11537 40220 11164 22773 21984 i
*> 204.84.32.0/20     146.57.255.241        2693    202      0 11537 81 11164 6939 27446 i
*> 216.54.48.0/24     146.57.255.241        2143    202      0 11537 40220 11164 22773 i
*> 216.54.49.0/24     146.57.255.241        2143    202      0 11537 40220 11164 22773 i
*> 216.146.50.0/24    146.57.255.241        2143    202      0 11537 40220 11164 6939 22299 i
*> 216.235.226.0/24   146.57.255.241        2143    202      0 11537 40220 11164 6939 26202 i
*> 216.235.226.0/24   146.57.255.241        2143    202      0 11537 40220 11164 6939 26202 i

And these are Google Global Cache Anycast addresses that probably shouldn't be in the R&E table, especially coming from Africa. Please note that I receive 104.237.191.0/24 via local peering with Google and was routing it to Africa until I reduced the local pref of these routes.

*> 104.237.175.0/24   146.57.255.241        2751     10      0 11537 36944 327687 36040 i
*  104.237.191.0/24   146.57.255.241        2751     10      0 11537 36944 327687 36040 i

I suppose some of these could be temporary issues, but I've seen many of these in the R&E table for a while now. So, could someone from Internet2 or GRNOC work with these connectors and international partners to clean up these issues? Even if that means Internet2 needs to filter some of these routes.

Once cleaned up, I'd like to recommend sanity filters to prevent the reoccurrence of these types of issues. Minimally I'd like to suggest that connectors should not be allowed to recirculate I2PX and ESNet routes into the R&E table, but I'd also like major commercial ISP to be included too. 

Thanks

--

===============================================
David Farmer              
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota  
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

--

===============================================
David Farmer              
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota  
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

--
Jeff Bartig
Interconnection Architect
Internet2  AS11164 / AS11537
+1-608-616-9908

--

===============================================
David Farmer              
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota  
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================