Internet2 Network Security SIG

["Schopis, Paul" <>]

Shannon,

It paid for 32 devices and some amount of scrubbing in the cloud. How it works is when enough traffic hits the appliance (that level is configurable) it rolls over to the cloud scrubbing service and then routes your clean traffic back over a GRE tunnel. So you get decide how much to allow. But it also looks like it would be an ideal role for I2 to play to address your concern commodity charges which of course opens the can of worms about it impacting the researchers.

 

The pilot testing indicated the appliance worked well with several Gbps, and I don’t think it ever hot the threshold to roll over to the cloud. I can try to get particulars if you’d like.

 

Be well,

Paul

 

 

From: Spurling, Shannon [mailto:]
Sent: Wednesday, October 21, 2015 10:32 AM
To: Schopis, Paul; Taylor, Scott J.; D'Angelo, Cas (Samuel); Steven Wallace;
Cc: Rob Vietzke; George Loftus; John Moore; Caroline Weilhamer
Subject: RE: [Security-WG] fast track for DDoS recommendations to Internet2, and a bit more...

 

I’ve noticed that a few of you are fine doing local scrubbing, but what do you do about the bandwidth consumption that you would be charged for, because that traffic is entering your network to be scrubbed? Also, that 2.1 Million could buy a few Arbor devices, but what kind of performance do you get on those?

 

 

Shannon Spurling

 

 

From: Schopis, Paul []
Sent: Wednesday, October 21, 2015 7:54 AM
To: Taylor, Scott J.; Spurling, Shannon; D'Angelo, Cas (Samuel); Steven Wallace;
Cc: Rob Vietzke; George Loftus; John Moore; Caroline Weilhamer
Subject: RE: [Security-WG] fast track for DDoS recommendations to Internet2, and a bit more...

 

Scott,

We had a lot of attacks on k-12 as well, the precipitating event was electronic testing state wide. Most of k-12 here uses NAT tas well, but we were able to mitigate some of impact by identifying characteristics of the troublesome traffic and rate limiting it. That way we didn’t kill the site and the evil doers were using little used port numbers so it had minimal impact on legit traffic. We even built a custom monitor so we could easily see it when it started. It was a high enough profile issue politically that the Ohio Dept of Ed gave a 2.1 million dollar grant and the Arbor solution was deployed. They won an RFP. It is a 3 year contract and the irony is as it was rolled out the attacks stopped. It did seem to work well during the pilot testing and the sites are connected to OARnet via 10G.

 

From: [] On Behalf Of Taylor, Scott J.
Sent: Tuesday, October 20, 2015 10:37 PM
To: Spurling, Shannon; D'Angelo, Cas (Samuel); Steven Wallace;
Cc: Rob Vietzke; George Loftus; John Moore; Caroline Weilhamer
Subject: Re: [Security-WG] fast track for DDoS recommendations to Internet2, and a bit more...

 

I’m curious to hear if anyone has had any success in building scrubbing type mitigation inside their networks?  The majority of the attacks we see are focused on our K12 environment and range from 1-20Gbps.  The K12 members often hide all their traffic behind a single NATed address and while we can blackhole traffic it still means they are either down or must renumber.  When they renumber we often see the attack follow them to the new IP.

 

We’ve been investigating building a scrubbing solution to mitigate volumetric attacks up to about 40Gbps.  Larger than that we’d blackhole or help broker redirecting their traffic to a mitigation service.  So far we’ve had poor success but haven’t had a lot of time to spend on this either.  We’ve done a lot of talking with vendors but only done a single POC, which failed to meet our requirements.

 

I’m starting to believe that the IU guys that are doing SCI-Flow (?) have the right model for DDoS mitigation as well as expressing elephant flows.  Why can’t when we detect these attacks, we program a controller to drop.  I’m also very curious to spend some more time with vendors on the BGP-Flowspec capabilities and maybe using something like that to drop traffic at our edge.  Based on what we’ve seen in CT I have to believe we could easily knock out the less sophisticated attacks.

 

-Scott

 

 

From: <> on behalf of "Spurling, Shannon" <>
Date: Tuesday, October 20, 2015 at 12:06
To: "D'Angelo, Cas (Samuel)" <>, Steven Wallace <>, "" <>
Cc: Robert Vietzke <>, George Loftus <>, John Moore <>, Caroline Weilhamer <>
Subject: RE: [Security-WG] fast track for DDoS recommendations to Internet2, and a bit more...

 

Is scrubbing something I2 wants to provide or contract for? I think the power to leverage the AL2S for temporary up-links into a scrubbing service that is a pay-as-you-go type offering could be powerful. There has been some trepidation on the part of certain members with making full use of the TR-CPS service, and then there is also the location limited membership of the TR-CPS peers. If the scrubbing service’s outside facing presence is sufficiently diverse and  present in the unwashed mass of the Internet, coming in through the backside with AL2S might be the big plus over layer3 tunneling. I think getting a large enough and diverse enough group of touch points into commodity Internet  to make the scrubbing work well would be hard for I2 without complete community participation. Maybe something Members would have to be willing to host a box that could draw dirty traffic in to be scrubbed and advertise those routes out to their Internet providers in some way. But that’s cazy… Right?

 

Personally, I’m not sold on scrubbing. Sometimes it’s best to scuttle the IP during the attack and adopt some edge based practices that let you have some flexibility at the edge. Some of the lamest (as far as target value or reason behind it) DDOS’s are enormous, and I don’t see any way to effectively scrub them out. Then there’s the camouflaged ones, where you would need something application or state aware to properly remove the bad traffic. That is very computationally expensive.

 

Shannon Spurling

 

 

From: [] On Behalf Of D'Angelo, Cas (Samuel)
Sent: Tuesday, October 20, 2015 7:37 AM
To: Steven Wallace;
Cc: Rob Vietzke; George Loftus; John Moore; Caroline Weilhamer
Subject: Re: [Security-WG] fast track for DDoS recommendations to Internet2, and a bit more...

 

Steve,

 

I like your list of requirements.  Should we say something about the pricing model we'd prefer? 

  small recurring charge with burst fee for scrubbing

  higher recurring charge with no additional fees

  some other model

 

Does Internet2 (and TR-CPS) provide the routing based tools today?  Should we start a separate project asking I2 to implement:

• RTBH with flow spec
• UTRS

maybe more???

 

Thanks.

 

Cas

 

---

From: <> on behalf of Steven Wallace <>
Sent: Friday, October 16, 2015 9:59 AM
To:
Cc: Rob Vietzke; George Loftus; John Moore; Caroline Weilhamer
Subject: [Security-WG] fast track for DDoS recommendations to Internet2, and a bit more...

 

[cc’ing Caroline for NTAC engagement]

 

 

I received a couple of volunteers (i.e. 2)  to participate in developing a fast-tracked set of recommendations to Internet2 concerning possible DDoS service offerings. Since this is going to be very light weight, I’m soliciting the entire group to weigh in.

 

I confirmed with Rob Vietzke that our charge is:

 

“Internet2 requests that the Security WG recommend a set of DDoS mitigation capabilities to be delivered using, or in conjunction with, the Internet2 network. The intent is to inform Internet2’s expeditious engagement with mitigation providers to on-board services.”

 

The idea is that the group would provide a list of DDoS mitigation capabilities Internet2 might offer. To expedite I2’s delivery of these capabilities, I2 is not seeking specific vendor recommendations. It’s also my understanding that such services may or may not be delivered via Net+ (as someone said expeditiously and via Net+ have not been proven to co-exist ;-).

 

I’m going to take this charge a bit farther and also solicit additional Internet2 operational capabilities, such as RTBH with BGP Flowspec and participation in UTRS.

 

 

I’ll prime the pump with some desired capabilities:

 

• scrubbing service (the ones that announce the campus’s IP space)
• web, and web-based application caching
• RTBH with flow spec
• UTRS
• DDoS detection

 

 

 

I think it also good to list any constraints that will help inform I2’s efforts.

 

Let’s use the list to contribute. Next Friday I’ll summarize and deliver to I2 are recommendations. This feels like real progress for this group.

 

thanks,

 

ssw