perfSONAR User Q&A and Other Discussion

[Purna Mididuddi <>]

Hello Mark,
Thanks a lot for taking the time to respond.

In the output I sent before, the offset for bare-metal is also drifting by 
about 15ms (10.046 to -4.279), with all the bare-metal advantages that it 
has. Note that this is to a stratum 1 server. 
Is this acceptable in general, for different type of perfsonar tests? If so, 
I wouldn't worry much about running the tests on VM which is also showing 
similar drift.

On bare-metal:
-----------------
root@HOST:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 *saturn.es.net   .CDMA.           1 u   57   64  177   24.547   10.046   
5.885
root@HOST:~#
root@HOST:~#
root@HOST:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 *saturn.es.net   .CDMA.           1 u   59   64  377   24.889   -4.279   
5.590

Thx,
Purna

-----Original Message-----
From: Mark Feit 
[mailto:]
 
Sent: Wednesday, December 14, 2016 7:19 AM
To: Purna Mididuddi 
<>;
 

Subject: Re: [perfsonar-user] Clock stability on VM

Purna Mididuddi writes:
>
> We are looking to run a perfsonar node on a VM, with dedicated NIC, 
> dedicated CPU cores and disabling VM swapping. Also we are running NTP 
> on the VM. Given these, can we assume clock on VM will be accurate?

Accuracy usually comes with a measurable figure, e.g. “within x seconds of 
the actual time.”  The value of x and the topological distance between the 
participants have a heavy influence on the correctness of latency 
measurements, so it’s important to understand the limitations.  For example, 
it’s possible to come up with a negative latency figure if the participants’ 
clocks are off in such a way that the packets look like they arrived before 
they departed.  NTP synced to random hosts on the Internet is usually good to 
within 1 ms and doesn’t have this problem if the hosts are more than 100 
miles apart; using local, stratum-1 servers gets that down to to 250 µs and 
25 miles.  There are ways to get lots more accurate than that, but it gets 
very expensive very quickly.

I look at measuring latency like I look at measuring the fuel economy in my 
car: small variations don’t really mean much.  Getting 29.7 miles per gallon 
on one tank when I usually get 30 isn’t cause for alarm, but if I suddenly 
get 15 when all other things were about equal, something’s probably wrong.  
On your network, even with the clock not being ultra-precise, you’re going to 
find normal ranges and deviations that indicate problems.  If latency between 
two points is historically 20 ms and it jumps to 50, something in the middle 
has changed for the worse.

> The "ntpq -p" output on VM shows the offset column is changing slightly 
> over a period.
> But same behavior is seen on bare-metal output too. So, how can we say 
> baremetal has more clock accuracy than VM?

One word:  hardware.  The operating system gets a regular stream of hardware 
interrupts that are used to keep the clock ticking.  On a bare-metal machine, 
those interrupts are generated by a circuit that does so with a minimum of 
external influences, usually just voltage and temperature.  If there are no 
factors on the system that prevent the kernel from servicing interrupts in a 
timely manner, the system clock will be no more accurate or precise than the 
hardware timer.  Any increments of time you see on 

VMs have no such hardware.  Their hosts do, but that interrupt has to pass 
through the hypervisor, which adds some unpredictable amount of latency 
between the interrupt’s arrival and the time it gets passed to the guests, 
who then have to process it and adjust their own clocks.  The steps you’ve 
taken (dedicated resources and disabling VM swapping) will help make sure the 
arriving virtual interrupt is processed quickly and that NTP can contribute 
to the clock’s long-term stability, but neither will make the hypervisor any 
more deterministic.  A busier hypervisor is likely to be worse about this 
than one that’s running a single guest.

Hope that helps.

--Mark