perfSONAR User Q&A and Other Discussion

[Andrew Gallo <>]

I certainly appreciate Dan M's championing on this topic, and as he 
mentioned, I would like to see PTP more widely supported in monitoring 
applications.

Certainly, in early 2016, PTP is an anomaly on campus networks, but I'm 
seeing a growing demand for more precise timing from things like Cisco's 
ACI (which keeps intra-data center fine grained packet statistics) to 
cellular rebroadcast over enterprise LANs (cellular handover requires 
very precise agreement on clocking).

Without going to far astray, I do think we'll see PTP become much more 
common on enterprise networks.  Oddly, one of the factors that led to 
Ethernet's success is that it was cheaper and easier to deploy 
specifically because it didn't contain the timing hardware that its TDM 
competitors had.  But Ethernet is becoming  the de facto transport for 
just about everything, and things like audio/video bridging and even 
vehicle borne Ethernet requires much more than NTP can deliver.  So we 
now have PTP and Sync-E, which gets us back to the same specs as SONET 
Minimum Clock.  The circle is nearly complete.

Are we early on in the high precision timing game?  Is this a 
chicken-and-egg problem?  Or is PTP something that will remain uncommon?



On 2/19/2016 4:07 PM, Magorian, Daniel F. wrote:
> Ok, Jason wrote me back privately saying similar things, and so I agree that 
> PTP and LAN use is out-of-scope for PerfSONAR.
>
> So I'll stop hammering on PerfSONAR to become a LAN tool that isn't in scope 
> for the project.  And in return, PerfSONAR folks should stop pretending it 
> can do stuff it can't, and firmly define their WAN scope to not lead people 
> new to measurement astray by implying that it can be used for LAN 
> measurement.  Someone should dump that misleading PTP statement and just 
> define the scope properly.  I suggest this rewrite put somewhere prominent:
>
> "PerfSONAR's primary mission is identifying network problems along paths between 
> WAN domains.  The latency in those paths tends to be large enough that NTP's 
> millisecond accuracy is sufficient for most testing.  Intra-campus LAN path 
> measurements needing sub millisecond timing should use other tools than PerfSONAR that 
> use PTP and are better suited to those tasks.   At some point the Performance Working 
> Group should form a task area to evaluate and recommend the use of appropriate LAN 
> latency measurement tools, such as smart SFP-based systems"
>
> > Because I think that pretty much nobody is on the same page as you are, but coming from you, it 
> > intrigues us...   What I would be very interested in are: "An Exposition Of The Joys Of 
> > Short Path Latency Analysis"  and  "How To Implement PTP Timing In An Academic 
> > Enterprise Network for OWAMP Discipline"
>
> Thanks. Well, I know Andrew Gallo of GWU gets where I'm coming from because 
> he was also asking for PTP, but anyway...     I think the main difference is 
> that I've been working with new smart SFPs and other people have not.  So 
> they may be unaware of how purpose-built cheap hardware has changed the whole 
> measurement landscape compared to more traditional software-based measurement 
> tools.  Call it part of the Internet of Things:  previously dumb stuff like 
> SFPs now acquiring smarts.
>
> Yeah, I could write some white papers and how-tos on it, but you know those 
> don't usually get read or paid attention to.
>
> What might be the right approach is a new LAN measurement task area within 
> the Performance Working Group, given that PerfSONAR is the endorsed task area 
> for WAN measurement.  Then those white papers could be the new LAN task area 
> charter, and folks like me who are more interested in the LAN side could sign 
> off this list and leave the WAN PerfSONAR fans in peace.  That task area 
> could then focus on smart SFPs and other new tech for LAN measurement, and 
> obviously people could tune into both if they measure both.   I think it 
> nicely fits the I2 mission to reinforce both LAN and WAN sides these days and 
> not just be all about WANs.
>
> The problem is, I have no idea how to do this.   I need help from someone 
> with an administrative bent to navigate that process...
>
> Dan
>
> -----Original Message-----
> From: 
>
>  
> [mailto:]
>  On Behalf Of Alan Whinery
> Sent: Friday, February 19, 2016 2:58 PM
> To: 
>
> Subject: Re: [perfsonar-user] feedback on PTP
>
> Dan,
>
> You can't have both:
> > > Dan in his talk today said  something like  "Well, anything below 1
> > > mS is basically instantaneous" .  Again, spoken with a strong WAN
> > > bias and little understanding of intra-campus paths.
>
> and
>
> > Sorry, without the ability to see below 1 mS, there is no such thing
> > > as "a general sense of campus network performance".
>
> Sorry, but I get value value from analyzing packet loss exclusively,
> which probably points out the same queuing-related problems that I
> assume you're looking for with latency.
>
> And assailing the perfSONAR project's foci is a little like writing your
> congressman, it has ceremonial flair, little value otherwise.
>
> What I would be very interested in are:
>
>         "An Exposition Of The Joys Of Short Path Latency Analysis"
>         and
>         "How To Implement PTP Timing In An Academic Enterprise Network     
>         For OWAMP Discipline"
>
> or existing links.
>
> Because I think that pretty much nobody is on the same page as you are,
> but coming from you, it intrigues us, because you say smart stuff, and
> there's much implicit "exercise for the reader" in your in-band comments
> that I've seen.
>
> -Alan
>
>
> On 2/19/2016 8:40 AM, Magorian, Daniel F. wrote:
> > Actually, surprise!  Smart SFP TWAMP reflectors (that also can sniff
> > packets for remote pcap and other useful things) are $300/ea, only
> > about 2x what the small nodes cost, and the central initiator that
> > times the TWAMP reflectors is also cheap.  As an SFP it's also super
> > easy to replace ordinary SFPs with.  I bought the two dozen and the
> > traffic initiator for < $5k, chump change.
> >
> > > small-footprint nodes can be deployed all over campus, to give us a
> > > general sense of network performance
> >
> > Sorry, without the ability to see below 1 mS, there is no such thing
> > as "a general sense of campus network performance".  Either you can
> > resolve your campus path latency, or you just have lines along the
> > bottom of your graphs at 1 mS, pretty worthless.
> >
> > Why settle for low resolution when you can have high for close to the
> > same cost?
> >
> > Dan
> > -----Original Message-----
> > From: Peter Gutierrez 
> > [mailto:]
> > Sent: Friday, February 19, 2016 1:13 PM
> > To: Magorian, Daniel F.; 
> > ;
> >  
> >
> > Cc: Andrew Gallo; 
> > ;
> >  Jason Zurawski
> > Subject: Re: [perfsonar-user] feedback on PTP
> >
> > On 02/19/2016 12:43 PM, Magorian, Daniel F. wrote:
> > > Dan and Jason, this discussion on PTP pretty much misses the point
> > > entirely,
> >
> >
> > In my opinion, accurate intra-campus latency measurement has value, but I
> > don't expect it from a cigarette-box server any more than I expect them to
> > provide 100Gbps interfaces for bandwidth testing. As long as these
> > low-cost, low-power, small-footprint nodes can be deployed all over
> > campus, to give us a general sense of network performance, it's a yoooge win
> > .
> >
> > Deploying expensive, pizza-box servers with appropriate hardware to do
> > more advanced latency/bandwidth measurements in the places I need it is a
> > worthwhile investment as well.
> >
> > PeterG
> >
> >
> > > https://www.perfsonar.net/deploy/timekeeping-with-ntp/
> >
> > > and demonstrates how software guys rarely see the need for/advantages
> > > of dedicated hardware.  Follow that logic to its conclusion and you can
> > > do away with small nodes altogether, and just create a centralized VM
> > > box with virtual measurement node instances cabled to wherever.  Or
> > > containers these days, which at least are lighter weight.
> >
> > > Obviously no one thinks that *synchronizing* PTP is cost-effective yet,
> > > as would be needed for OWAMP.
> >
> > > The point you're missing is that TWAMP, while it does traverse the path
> > > twice, goes and comes back to the same PTP initiator.  Therefore it has
> > > no need to sync to anything else, and is accurate internally to ~1 uS
> > > so intra-campus paths can be measured.
> >
> > > Since you're showing no interest in measuring such intra-campus paths,
> > > you need to stop saying "perfsonar world domination" and change it to
> > > "perfsonar inter-campus WAN world domination".  And if that's your
> > > mission and the scope of the project, software NTP-based timing to mS
> > > works fine.
> >
> > > Just know that you're making the project unusable for an enormous new
> > > intra-campus market, due to WAN-oriented bias.
> >
> > > Dan in his talk today said  something like  "Well, anything below 1 mS
> > > is basically instantaneous" .  Again, spoken with a strong WAN bias and
> > > little understanding of intra-campus paths.  If your north-south
> > > core-to-edge campus traffic is talking 1000 uS, you have serious
> > > performance issues that really need to be looked into and probably have
> > > users beating you up daily.  Which normally everyone blames on
> > > firewalls and repeats that tired litancy without bothering to actually
> > > measure, because they can't without the right tools.
> >
> > > In reality good Cisco or Palo Alto campus firewalls introduce ~ 50 -
> > > 100 uS latency when working properly, as does your IPS.  I know,
> > > because I measure ours and watch for latency spikes on the cacti
> > > graphs.
> >
> > > My overall point is, you could extend Perfsonar's usability to
> > > intra-campus path measurement relatively easily with some architectural
> > > changes, but nobody wants to because of WAN bias.
> >
> > > Sigh.
> >
> > > Dan Magorian

--
________________________________
Andrew Gallo
The George Washington University

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature