perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Purna Mididuddi writes:
>
> We are looking to run a perfsonar node on a VM, with dedicated NIC, 
> dedicated
> CPU cores and disabling VM swapping. Also we are running NTP on the VM. 
> Given these,
> can we assume clock on VM will be accurate? 

Accuracy usually comes with a measurable figure, e.g. “within x seconds of 
the actual time.”  The value of x and the topological distance between the 
participants have a heavy influence on the correctness of latency 
measurements, so it’s important to understand the limitations.  For example, 
it’s possible to come up with a negative latency figure if the participants’ 
clocks are off in such a way that the packets look like they arrived before 
they departed.  NTP synced to random hosts on the Internet is usually good to 
within 1 ms and doesn’t have this problem if the hosts are more than 100 
miles apart; using local, stratum-1 servers gets that down to to 250 µs and 
25 miles.  There are ways to get lots more accurate than that, but it gets 
very expensive very quickly.

I look at measuring latency like I look at measuring the fuel economy in my 
car: small variations don’t really mean much.  Getting 29.7 miles per gallon 
on one tank when I usually get 30 isn’t cause for alarm, but if I suddenly 
get 15 when all other things were about equal, something’s probably wrong.  
On your network, even with the clock not being ultra-precise, you’re going to 
find normal ranges and deviations that indicate problems.  If latency between 
two points is historically 20 ms and it jumps to 50, something in the middle 
has changed for the worse.

> The "ntpq -p" output on VM shows the offset column is changing slightly 
> over a period.
> But same behavior is seen on bare-metal output too. So, how can we say 
> baremetal has
> more clock accuracy than VM?

One word:  hardware.  The operating system gets a regular stream of hardware 
interrupts that are used to keep the clock ticking.  On a bare-metal machine, 
those interrupts are generated by a circuit that does so with a minimum of 
external influences, usually just voltage and temperature.  If there are no 
factors on the system that prevent the kernel from servicing interrupts in a 
timely manner, the system clock will be no more accurate or precise than the 
hardware timer.  Any increments of time you see on 

VMs have no such hardware.  Their hosts do, but that interrupt has to pass 
through the hypervisor, which adds some unpredictable amount of latency 
between the interrupt’s arrival and the time it gets passed to the guests, 
who then have to process it and adjust their own clocks.  The steps you’ve 
taken (dedicated resources and disabling VM swapping) will help make sure the 
arriving virtual interrupt is processed quickly and that NTP can contribute 
to the clock’s long-term stability, but neither will make the hypervisor any 
more deterministic.  A busier hypervisor is likely to be worse about this 
than one that’s running a single guest.

Hope that helps.

--Mark