perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

ELGER, NATHAN writes:

 

We've recently purchased a few 100G hosts to use for perfsonar testing across our campus 100G network as well as external perfsonar hosts in the R&E space.

…

My issue is that with any scheduled throughput tests to 100G hosts, such as TACC, we are only seeing around 20gbit forward with very asymmetrical speeds back to us- around 5gbit. Some tests to 40gbit hosts at other institutions have even worse asymmetry- a test to gatech has about 200mbit back to us and we should both be going over I2. Even with a scheduled throughput test between my 2 perfsonar hosts in the same switch, I only see ~15gbit between them. I have scripted 8 parallel iperf3 tests and although there are a lot of retransmissions, I do see a total of around 50-60gbit for the testing period of 30 seconds.

 

Iperf3 was designed for a specific use case and has architectural limitations that limit the speed a single invocation can achieve.  Iperf2 is multithreaded and does better but has some warts of its own.  (There’s some useful discussion about that here:  https://github.com/esnet/iperf/issues/289.)  Give iperf2 a try and see how it does for you.

 

SDSC has been doing some experimentation at high speeds with Ethr and is seeing good results.  We’ve developed a pScheduler plugin for it that will ship with 4.3 on a not-installed-by-default basis.  (You’ll be able to install it with “yum install pscheduler-tool-ethr”.)  Ethr’s Achilles’ heel is that it was written in Go and is subject to stop-the-world events when the garbage collector runs.  That has some effect on the results because it’s not making traffic during those times.

 

Given what we know in general about how the throughput tools behave, the sub-25 Gbps measurements are likely valid.  The problem here is that a system that hasn’t been vetted for performance is measuring a network whose characteristics are unknown.  Even back-to-back across a switch isn’t good if the switch isn’t known to be capable of passing the traffic it’s being sent.  The best thing to do is directly connect two of the systems and see what you can get out of them before measuring anything else.

 

--Mark