perfSONAR User Q&A and Other Discussion

[Aaron Brown <>]

Hey Pedro,

On Dec 10, 2012, at 2:56 PM, Pedro Queirós <> wrote:

Hello Joe,

thank you for your answer. So what you're saying is that

basically this can happen or not, depending on HW/OS

combinations, which are not fully understood?

I was hoping to replace the proprietary system in use, by using

perfSonar toolkit, but these findings may not allow it.

Anyway, I'd appreciate more insight on this issue, such as examples

of solutions currently in use (I mean hardware and operating systems)

that do not produce these artifacts.

I guess it depends on what you're using the maximum value for. For these kinds of measurements, we've found that doing the 95th percentile gives an accurate display of jitter/latency seen between hosts without being affected by transient host-based issues (generally affected by scheduling or I/O).

Beyond that, the maximum values that you're seeing seem oddly high (e.g. 200ms). Do you have anything else running on that host? Especially something that does I/O.

Cheers,

Aaron

Kind Regards,

Pedro Queirós

On Mon, Dec 10, 2012 at 7:35 PM, Joe Metzger <> wrote:

Pedro,
I have spent a lot of time on this issue in the past.

There are a number of hardware based solutions that go to great
lengths to make very accurate NIC to NIC measurements. Most of these
include carefully engineered strategies to prevent anomalous
measurements from being taken, and/or filter them out when they happen.

The OWAMP software used in most PerfSONAR implementations takes a
statistical approach to capture application to application latency.
It uses general purpose hardware, and general purpose operating systems
and so the measurements include both OS & network anomalies. It doesn't
discard obvious anomalies caused by measurement artifacts like the
other systems I have seen.

There are advantages and disadvantages of each approach, but I think
it is important to start from a position where you understand these
differences.

All of the big anomalies that I have tracked down were due to OS
issues.  It is fairly common to see packets spending 100ms or more
waiting for interrupts to be serviced. The frequency of packets
taking this slow path through the kernel grow significantly as
the load on the boxes grow.  Changing OS (Linux vs FreeBSD), as
well as OS version changes also impact this, but I haven't seen a
solution that eliminates it.

--Joe

On Dec 10, 2012, at 11:00 AM, Pedro Queirós wrote:

> Hi there,
>
> we're using OWAMP to measure the link between two sites we have,
> using two dedicated machines.
> In another two machines we have a proprietary solution to do the same
> task.
>
> We're finding that the average results are nearly identical, but when
> using OWAMP, the max delay that's show in the graph has several
> spikes that are not displayed when using the proprietary solution. In
> fact, the proprietary solution only shows max delays values close to
> the average, that is, around 3~4 ms.
>
> I was wondering if anyone has had the chance to run OWAMP side by
> side with other similar tools and compare the results. We can't find any
> reason why those spikes appear in the graphs in OWAMP.
> They also appear more frequently in IPv4 measurements vs IPv6
> measurements.
>
> I'll attach screenshots so you can hopefully understand better what I
> mean. As you can see, in IPv6, the spikes are much more rare.
>
> I'm hoping to find a reason for these spikes, because nothing else
> (CPU load, network congestion) justifies these spikes in the
> measurements - and they don't appear in the measurements we have
> in place using a proprietary solution.
>
>
> Kind Regards,
> Pedro Queirós

> <ipv4_backward_max.png><ipv4_forward_max.png><ipv4_min.png><ipv6_backward_max.png><ipv6_forward_max.png><ipv6_min.png>

Joe Metzger