perfSONAR User Q&A and Other Discussion

["Magorian, Daniel F." <>]

We probably should be clear and not conflate time sync protocols and 
measurement protocols.  Obviously you can't measure better than your time is 
synced, unless you punt on the sync problem and measure back to the 
initiating box with a measurement protocol like TWAMP, which works with LANs 
since the distances are short.  GPS needs a connection to an outside antenna 
right, so doesn't that limit its use in time syncing  to places you can run 
coax up to a roof?  

> That said, I'm not too sure what I am going to see in a LAN environment 
> other than what I expect, tiny buffers periodically filling up and 
> middleboxes that do not improve performance.

Sure and what's all of networking measurement except watching buffers small 
or large filling and emptying on line cards and timing packets in between 
them?  Doesn't mean it isn't worth measuring same as WAN performance, if you 
can.  In fact it's useful to see the classic small buffer bracket verticals 
like I sent examples of before in your paths, to know how much they're 
actually hurting you at small and large packet sizes.

It's easy to say "middleboxes that don't improve performance"  but it's worth 
measuring to see how much they DO impact performance.    Again we can stand 
on our high horses and put down those middleboxes and espouse end-to-end 
transparency.   But the bottom line is in today's campus security 
environment, dissing firewalls and the like is as much a waste of time as 
writing your Congressman, as Alan said.  "Ain't gonna go away".   Sure, 
science DMZs are fine, but what about average users' performance trying to do 
FTPs from their desk machines?  It really helps to triage what's happening 
with the underlying network using measurement, from what's happening at the 
application layer on hosts.  If you have the right tools and can.

Dan

-----Original Message-----
From: Dale W. Carder 
[mailto:]
 
Sent: Friday, February 19, 2016 4:33 PM
To: Magorian, Daniel F.
Cc: Alan Whinery; 
;
 Jason Zurawski; Andrew Gallo
Subject: Re: [perfsonar-user] feedback on PTP


You don't necessarily need to use PTP on your endpoints, as you could 
probably discipline them individually to GPS.  I was thinking of
experimenting along these lines with some boxes we have w/ rubidium
oscillators, but I can't seem to get a correct build environment offhand 
to get owamp on there (granted, I spent like 3 minutes).

That said, I'm not too sure what I am going to see in a LAN environment
other than what I expect, tiny buffers periodically filling up and 
middleboxes that do not improve performance.  So, I am pretty curious
about the use-case (regardless of PerfSONAR) because I would like to 
know what I'm missing here.

Dale


Thus spake Magorian, Daniel F. 
()
 on Fri, Feb 19, 2016 at 09:07:25PM +0000:
> Ok, Jason wrote me back privately saying similar things, and so I agree 
> that PTP and LAN use is out-of-scope for PerfSONAR.  
> 
> So I'll stop hammering on PerfSONAR to become a LAN tool that isn't in 
> scope for the project.  And in return, PerfSONAR folks should stop 
> pretending it can do stuff it can't, and firmly define their WAN scope to 
> not lead people new to measurement astray by implying that it can be used 
> for LAN measurement.  Someone should dump that misleading PTP statement and 
> just define the scope properly.  I suggest this rewrite put somewhere 
> prominent:
> 
> "PerfSONAR's primary mission is identifying network problems along paths 
> between WAN domains.  The latency in those paths tends to be large enough 
> that NTP's millisecond accuracy is sufficient for most testing.  
> Intra-campus LAN path measurements needing sub millisecond timing should 
> use other tools than PerfSONAR that use PTP and are better suited to those 
> tasks.   At some point the Performance Working Group should form a task 
> area to evaluate and recommend the use of appropriate LAN latency 
> measurement tools, such as smart SFP-based systems"
> 
> > Because I think that pretty much nobody is on the same page as you are, 
> > but coming from you, it intrigues us...   What I would be very interested 
> > in are: "An Exposition Of The Joys Of Short Path Latency Analysis"  and  
> > "How To Implement PTP Timing In An Academic Enterprise Network for OWAMP 
> > Discipline"
> 
> Thanks. Well, I know Andrew Gallo of GWU gets where I'm coming from because 
> he was also asking for PTP, but anyway...     I think the main difference 
> is that I've been working with new smart SFPs and other people have not.  
> So they may be unaware of how purpose-built cheap hardware has changed the 
> whole measurement landscape compared to more traditional software-based 
> measurement tools.  Call it part of the Internet of Things:  previously 
> dumb stuff like SFPs now acquiring smarts.
> 
> Yeah, I could write some white papers and how-tos on it, but you know those 
> don't usually get read or paid attention to.
> 
> What might be the right approach is a new LAN measurement task area within 
> the Performance Working Group, given that PerfSONAR is the endorsed task 
> area for WAN measurement.  Then those white papers could be the new LAN 
> task area charter, and folks like me who are more interested in the LAN 
> side could sign off this list and leave the WAN PerfSONAR fans in peace.  
> That task area could then focus on smart SFPs and other new tech for LAN 
> measurement, and obviously people could tune into both if they measure 
> both.   I think it nicely fits the I2 mission to reinforce both LAN and WAN 
> sides these days and not just be all about WANs.
> 
> The problem is, I have no idea how to do this.   I need help from someone 
> with an administrative bent to navigate that process...
> 
> Dan
> 
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Alan Whinery
> Sent: Friday, February 19, 2016 2:58 PM
> To: 
> 
> Subject: Re: [perfsonar-user] feedback on PTP
> 
> Dan,
> 
> You can't have both:
> >> Dan in his talk today said  something like  "Well, anything below 1
> >> mS is basically instantaneous" .  Again, spoken with a strong WAN
> >> bias and little understanding of intra-campus paths.
> 
> and
> 
> > Sorry, without the ability to see below 1 mS, there is no such thing
> >> as "a general sense of campus network performance".
> 
> Sorry, but I get value value from analyzing packet loss exclusively,
> which probably points out the same queuing-related problems that I
> assume you're looking for with latency.
> 
> And assailing the perfSONAR project's foci is a little like writing your
> congressman, it has ceremonial flair, little value otherwise.
> 
> What I would be very interested in are:
> 
>       "An Exposition Of The Joys Of Short Path Latency Analysis"
>       and
>       "How To Implement PTP Timing In An Academic Enterprise Network  
>       For OWAMP Discipline"
> 
> or existing links.
> 
> Because I think that pretty much nobody is on the same page as you are,
> but coming from you, it intrigues us, because you say smart stuff, and
> there's much implicit "exercise for the reader" in your in-band comments
> that I've seen.
> 
> -Alan
> 
> 
> On 2/19/2016 8:40 AM, Magorian, Daniel F. wrote:
> > Actually, surprise!  Smart SFP TWAMP reflectors (that also can sniff
> > packets for remote pcap and other useful things) are $300/ea, only
> > about 2x what the small nodes cost, and the central initiator that
> > times the TWAMP reflectors is also cheap.  As an SFP it's also super
> > easy to replace ordinary SFPs with.  I bought the two dozen and the
> > traffic initiator for < $5k, chump change.
> > 
> >> small-footprint nodes can be deployed all over campus, to give us a
> >> general sense of network performance
> > 
> > Sorry, without the ability to see below 1 mS, there is no such thing
> > as "a general sense of campus network performance".  Either you can
> > resolve your campus path latency, or you just have lines along the
> > bottom of your graphs at 1 mS, pretty worthless.
> > 
> > Why settle for low resolution when you can have high for close to the
> > same cost?
> > 
> > Dan
> > -----Original Message-----
> > From: Peter Gutierrez 
> > [mailto:]
> >  
> > Sent: Friday, February 19, 2016 1:13 PM
> > To: Magorian, Daniel F.; 
> > ;
> >  
> > 
> > Cc: Andrew Gallo; 
> > ;
> >  Jason Zurawski
> > Subject: Re: [perfsonar-user] feedback on PTP
> > 
> > On 02/19/2016 12:43 PM, Magorian, Daniel F. wrote:
> >> Dan and Jason, this discussion on PTP pretty much misses the point
> >> entirely,
> > 
> > 
> > In my opinion, accurate intra-campus latency measurement has value, but I
> > don't expect it from a cigarette-box server any more than I expect them to
> > provide 100Gbps interfaces for bandwidth testing. As long as these
> > low-cost, low-power, small-footprint nodes can be deployed all over
> > campus, to give us a general sense of network performance, it's a yoooge 
> > win
> > .
> > 
> > Deploying expensive, pizza-box servers with appropriate hardware to do
> > more advanced latency/bandwidth measurements in the places I need it is a
> > worthwhile investment as well.
> > 
> > PeterG
> > 
> > 
> >> https://www.perfsonar.net/deploy/timekeeping-with-ntp/
> > 
> >> and demonstrates how software guys rarely see the need for/advantages
> >> of dedicated hardware.  Follow that logic to its conclusion and you can
> >> do away with small nodes altogether, and just create a centralized VM
> >> box with virtual measurement node instances cabled to wherever.  Or
> >> containers these days, which at least are lighter weight.
> > 
> >> Obviously no one thinks that *synchronizing* PTP is cost-effective yet,
> >> as would be needed for OWAMP.
> > 
> >> The point you're missing is that TWAMP, while it does traverse the path
> >> twice, goes and comes back to the same PTP initiator.  Therefore it has
> >> no need to sync to anything else, and is accurate internally to ~1 uS
> >> so intra-campus paths can be measured.
> > 
> >> Since you're showing no interest in measuring such intra-campus paths,
> >> you need to stop saying "perfsonar world domination" and change it to
> >> "perfsonar inter-campus WAN world domination".  And if that's your
> >> mission and the scope of the project, software NTP-based timing to mS
> >> works fine.
> > 
> >> Just know that you're making the project unusable for an enormous new
> >> intra-campus market, due to WAN-oriented bias.
> > 
> >> Dan in his talk today said  something like  "Well, anything below 1 mS
> >> is basically instantaneous" .  Again, spoken with a strong WAN bias and
> >> little understanding of intra-campus paths.  If your north-south
> >> core-to-edge campus traffic is talking 1000 uS, you have serious
> >> performance issues that really need to be looked into and probably have
> >> users beating you up daily.  Which normally everyone blames on
> >> firewalls and repeats that tired litancy without bothering to actually
> >> measure, because they can't without the right tools.
> > 
> >> In reality good Cisco or Palo Alto campus firewalls introduce ~ 50 -
> >> 100 uS latency when working properly, as does your IPS.  I know,
> >> because I measure ours and watch for latency spikes on the cacti
> >> graphs.
> > 
> >> My overall point is, you could extend Perfsonar's usability to
> >> intra-campus path measurement relatively easily with some architectural
> >> changes, but nobody wants to because of WAN bias.
> > 
> >> Sigh.
> > 
> >> Dan Magorian
> > 
> > 
> > 
> > 
>