NTAC Peering Working Group

[Rob Vietzke <>]

For background for the larger group on the Deepfield/Netflow Issue… Wading in here in hopes we might get some discussion that would help inform and prompt further discussion :-).

Internet2 brought in Deepfield, a network analytics engine that crunches netflow, dns, snmp and other data, to help understand how the backbone was being used an answer the long time campus and regional question of “how is my campus using the Internet2 network”. Deepfield does a pretty good job of crunching the data it collects and presenting usage patterns by Internet2 member, cloud provider, AS, router, geography, etc. With the addition of federated ID support, Deepfield also can create a “window” into the aggregate dataset that allows a campus administrator to see its own data (and who that campus is talking too.)  That part is working and seems to be valued by the campuses that are using it. Internet2 is also using it internally to understand how the network is being used and what growth patterns are.

Their has been interest by regional networks in access to the data to see how they are using their Internet2 connections, and Internet2 has piloted that access with a number of regionals. Multihoming of some campuses to multiple regionals, state networks sitting behind regionals and other technical complexities of our community and how the engine crunches the data and presents that data have made the data in that view less than perfect, but still valuable in many cases. 

Technical imperfections aside, Internet2 has a long standing privacy policy that essentially says we won’t share one member’s usage patterns or data with another member and we will do our best to protect the privacy of individual community members and projects. This is explicit in the netflow anonymization policy that the NTAC has helped edit over time. The larger expectations around more general privacy of one project to another have not been as well documented, but we continue to hear from individual community leaders/campuses who feel protecting privacy is important.   (By example: “I don’t mind people seeing an aggregate spike in my total traffic hit the network, but I don’t think campus X should be able to know that my Campus Y and our partner Z are moving 3 pB of genomics files every Monday, Wednesday and Friday or that we move gigabytes of data to/from a certain commercial partner")

In the case that I believe Chris is referring to, several regional had asked not only to be able to access the data for internal technical/planning efforts, but also to be able to use and present that data in conversations with their members (IE, where more than one member might see another members’ data or details about interactions between members). This use case ran up against the privacy intentions of not sharing one members data with another. Where we stand now is that Internet2 allows regionals access to Deepfield for their own internal planning/access, but there is an NDA that essentially states the data should not be used outside the regional.

There are certainly instances where sharing the usage patterns more broadly would benefit the community, but it is hard to technically provision a system that can also meet the privacy intentions. So, we are currently in a less than perfect state where not every need is met. 

Thoughts on where we go, either on list, or in the FTW setting would be very much appreciated.

Rob

From: <> on behalf of "Spears, Christopher M." <>
Date: Tuesday, August 18, 2015 at 4:45 PM
To: Matt Zekauskas <>
Cc: Michael Lambert <>, "" <>, Ryan Nobrega <>
Subject: Re: Viability of a P&R FTW

Any change in the NDA?  I know that held some people back.

On Aug 18, 2015, at 4:14 PM, Matthew J Zekauskas <> wrote:

With respect to DeepField: We've had some personnel turnover that I think has helped give the impression of fade, but it still is up and running.   My suspicion is that it has been more useful to universities than aggregators.   If you have feedback, I think it would be welcome.   Feel free to send to myself or to Ryan Nobrega (cc'd, not on this list currently) or .

--Matt

On 8/18/15 1:57 PM, Michael H Lambert wrote:

>Internet2 Tools:

>- Router-proxy

>- SNAPP

>- Deepfield (w/NDA)

Should Deepfield be on the list?  My impression is that after great initial hoopla, it has at best faded into the background.  Or is this feeling just caused by the lack of follow-up by Internet2 with the regionals?