perfSONAR User Q&A and Other Discussion

[Tim Chown <>]

Hi Mark,

On 14 Aug 2023, at 21:07, Mark Feit <> wrote:

Tim Chown writes:

But now PTP is rather more mature, and support in devices better, and most interestingly new Linux distributions include chronyd (which supports hardware timestamps) rather than ntpd.

Our interest would be in enchanted accuracy for one-way delay measurements, and thus the capability being added to owping.

 

The discussion that spurred the addition of the PTP item to the FAQ took place at TechEx 2015, so the topic is ripe for an update.  Support for PTP is much better than it was then, but a lot of the same problems remain.  As I see it, there are two:

 

One is time discipline in general.  PTP as a time source for Chrony will help to some degree but, not having access to any systems that can use it, I have no data on how much compared to, say, NTP at stratum 1.  Even if the system is able to grab high-accuracy time, the operating environment isn’t real-time and will dilute it by an unknown, very-jittery quantity.  The figures I’ve seen put it at as much as 100 µs in userland plus 20 µs in the kernel and device before it hits the wire.

Agreed, and a first useful step would be getting some practical deployment experience of sync behaviour from two perfSONAR servers who have a PTP-based source.  

The good news is that PTP-capable NICs can cut all of that out by applying the timestamp to outgoing and incoming frames in hardware just inboard of the PHY.  Microsecond accuracy opens up a lot of possibilities for use on shorter links and better accuracy for longer ones.  Using that would be a big step forward but brings us to the second problem:  the software.

 

UMich did some initial scouting some years back and found out that the mechanics of getting packets timestamped and retrieving them aren’t complicated.  A worthwhile first step would be to develop a bare-bones tool that can use those mechanisms to do single one- and two-way measurements as a proof of concept.

 

Making this work with [OT]WAMP opens some cans of worms, one of which is backward compatibility.  Senders should continue applying a software timestamp and error estimate (EE) for the benefit of receivers that don’t have the ability to detect a kernel- or hardware-applied timestamp in an incoming packet.  Substituting the timestamp is the easy part.  The OWAMP protocol doesn’t have provisions for a secondary EE to be stashed in it, which means a receiver using the EE in the protocol will get the software EE, which is the wrong one for when a kernel/hardware timestamp is available.  Similarly, if an EE for an available kernel/hardware timestamp was to be used in the packet in place of the software EE, receivers using the software timestamp would get the wrong value.  Making life even more fun, the protocol has no provisions for versioning, so there’s no way to support an expanded protocol or for sessions to be negotiated at a particular version.  It might be time to revise the OWAMP protocol to accommodate these things, but that’s a much-bigger discussion than this.

There are certainly many devils in the detail here, and I look forward to discussing them in the perfSONAR meeting next week. 

I’ve noted a few NRENs deploying PTP, so this question is only likely to come up a little more often. Some will use PTP to improve their NTP service reliance, others for SIEM, but better monitoring granularity is certainly of interest, perhaps moreso as streaming telemetry is used more commonly.

Disclaimer:  This project is not on the perfSONAR team’s radar or roadmap and there is currently no estimate for when work on it would start or be completed.

Noted, but there’s some things we can start now, like updating the text in the documentation to explain the above, and the current consensus.  

Tim