Internet2 Network Security SIG

[Pete Siemsen <>]

As John Hernandez wrote, the FRGP has been through the uRPF deployment process suggested by Grover. It has worked well so far. Our process was

1. Get customers to agree that this is worth doing, to be good Internet citizens. Management was involved.

. Having the FRGP do it is easier that doing it yourselves

. We plan to put uRPF on all interfaces - nobody gets to opt-out

. We plan to handle special cases with fail-filter rules

. You'll have plenty of time to stop leaking the packets that we'll eventually block

. Lots of openness, warnings, scheduled changes, no surprises, changes can be backed-out, and "we'll work with you" on it

. Will take several months, but there will be an agreed deadline for when we'll begin blocking packets

2. Implement uRPF feasible on our "customer" Juniper interfaces, with a fail filter that simply Syslogs and Accepts all traffic

3. Process the syslog messages to produce a webpage for customer perusal. Perl program runs every day, creates a webpage that shows statistics for past 24 hours. Webpage lists all customer interfaces that have suspicious packets. Attached is a screenshot. It's a friendly wall-of-shame - customers see how they are doing, how they relate to others. You only show up on this page if you leaked some packets. Webpage isn't available to non-FRGP folks.

4. For each leaky customer, there's a second webpage showing aggregated statistics about the leaked packets. Attached is a sample. This gives customers information to fix their leaks. They can track down and fix the root of the problem. They can implement egress ACLs. They can do nothing. Up to them.

5. Some of the leaks are "legitimate". Like, some customer edge routers generate ICMP response packets using source addresses from an interface other than the one you might expect. We worked with customers to define fail-filters that explicitly allow these IP addresses. We generalized this, basically letting customers tell us a whitelist of IPs that that we should accept. We'd change the whitelist, 24 hours later the webpage showed the results, then the customer would request us to change the whitelist again, etc.

6. To avoid any hairy flag day, we defined a schedule. We turned on blocking for 3 customers at a time. So on a Tuesday we did 3, then on a Thursday we did 3 more, etc. The schedule was published in advance, and we warned customers a few days before their day. Many customers didn't pay attention until they were warned. Some customers didn't care. There weren't that many issues.

The FRGP has about 30 "customers". Some were already clean. Some cleaned themselves up. Some needed us to whitelist a few addresses to get clean. Some didn't do anything, and just let us start dropping their leaked packets. Some found problems in their networks, using the packet information.

I think this process worked for everyone. Customers were able to expend as much effort as they liked to fix their problems. Many still leak packets, but they know what they are and let us block them. Many, many fewer packets (millions/24hr) fail the uRPF checks now than when we started.

Please let me know if you'd like the Perl code.

-- Pete

Attachment: Screen Shot 2017-11-07 at 2.32.18 PM.png
Description: PNG image

Attachment: Screen Shot 2017-11-07 at 2.37.01 PM.png
Description: PNG image