perfSONAR User Q&A and Other Discussion

["Fedorka, Shayne" <>]

Thanks for all of the replies. I was able to get much better results running parallel streams with iperf2. Iperf3 performs significantly worse.

 

[perfsonar-100g-a ~]$ iperf -c 172.16.10.20 -i 1 -w 32M -P 4

------------------------------------------------------------

Client connecting to 172.16.10.20, TCP port 5001

TCP window size: 64.0 MByte (WARNING: requested 32.0 MByte)

------------------------------------------------------------

[  6] local 172.16.10.10 port 47310 connected with 172.16.10.20 port 5001

[  3] local 172.16.10.10 port 47306 connected with 172.16.10.20 port 5001

[  5] local 172.16.10.10 port 47312 connected with 172.16.10.20 port 5001

[  4] local 172.16.10.10 port 47308 connected with 172.16.10.20 port 5001

[ ID] Interval       Transfer     Bandwidth

[  6]  0.0- 1.0 sec  2.44 GBytes  21.0 Gbits/sec

[  3]  0.0- 1.0 sec  2.91 GBytes  25.0 Gbits/sec

[  5]  0.0- 1.0 sec  2.95 GBytes  25.4 Gbits/sec

[  4]  0.0- 1.0 sec  1.80 GBytes  15.5 Gbits/sec

[SUM]  0.0- 1.0 sec  10.1 GBytes  86.8 Gbits/sec

[  6]  1.0- 2.0 sec  2.97 GBytes  25.5 Gbits/sec

[  3]  1.0- 2.0 sec  2.92 GBytes  25.1 Gbits/sec

[  5]  1.0- 2.0 sec  3.00 GBytes  25.8 Gbits/sec

[  4]  1.0- 2.0 sec  2.60 GBytes  22.4 Gbits/sec

[SUM]  1.0- 2.0 sec  11.5 GBytes  98.8 Gbits/sec

 

[perfsonar-100g-a ~]$ iperf3 -c 172.16.10.20 -i 1 -w 32M -P 4

Connecting to host 172.16.10.20, port 5201

[  5] local 172.16.10.10 port 47916 connected to 172.16.10.20 port 5201

[  7] local 172.16.10.10 port 47918 connected to 172.16.10.20 port 5201

[  9] local 172.16.10.10 port 47920 connected to 172.16.10.20 port 5201

[ 11] local 172.16.10.10 port 47922 connected to 172.16.10.20 port 5201

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd

[  5]   0.00-1.00   sec  1.00 GBytes  8.63 Gbits/sec    0   5.32 MBytes      

[  7]   0.00-1.00   sec  1.00 GBytes  8.62 Gbits/sec    0   4.25 MBytes      

[  9]   0.00-1.00   sec  1.00 GBytes  8.63 Gbits/sec    0   4.40 MBytes      

[ 11]   0.00-1.00   sec  1.00 GBytes  8.62 Gbits/sec    0   5.38 MBytes      

[SUM]   0.00-1.00   sec  4.02 GBytes  34.5 Gbits/sec    0            

 

--

Shayne Fedorka

Network Engineer | NREL

 

From: Mark Feit <>
Date: Friday, June 18, 2021 at 8:40 AM
To: Tim Chown <>, Eli Dart <>
Cc: "Fedorka, Shayne" <>, "" <>
Subject: Re: [perfsonar-user] 100G perfSONAR Performance

 

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Tim Chown writes:

 

Our experience is that iperf2 is more ‘friendly’ for higher throughput as it seems a little smarter on how it distributes the multiple streams to its processors, where iperf3 needs additional parameters to be set.

My understanding (and Bruce can correct me if I’m wrong) is that while it can be made to work for this use case, iperf3 wasn’t designed with it in mind.

 

perfSONAR also supports Ethr for throughput testing as of 4.3.0, which we found to perform very well as an alternative to either iperf version, albeit using more CPU.

If you can live without iperf3’s more-exotic features, iperf2 and Ethr  are the paths of least resistance for the time being.  Ethr’s only Achilles’ heel is that it was written in Go, whose garbage collection is tuned for latency at the expense of throughput.  This produces some observable blips in how much traffic it can make, but it was still capable of getting north of 90 Gb/s.  There is a new 1.x version of the program that we haven’t evaluated yet but will probably appear in the feature release of perfSONAR following 4.4.

If the current suite of tools in perfSONAR is inadequate, Ethr came about through a collaboration between the perfSONAR team and SDSC.  We’d be happy to work with anyone who’s found or developed something better and has a good understanding of how to use it.  This includes tools that do L2 measurements.

This is a good time to start thinking about how to measure at above 100 Gb/s.  Multi-400 Gb/s links are rapidly becoming a thing on backbones.

 

Some best practice guidance for 100G throughput tests would be useful.

Tuning the machine per the NIC vendor’s guidance should get you most of the way there.  In general, four cores seems to be the minimum number to get a good run in and you reach the point of diminishing returns at six.

One thing I can’t stress enough is that any evaluation of how these tools perform on your systems needs to be done in a lab setting on two systems with back-to-back interfaces.  Putting a network, especially a WAN, between them will get measurements of how the testers perform across the network, not how well the testers do.  (The first segment of my TechEXtra perfSONAR Day talk covers the principles of good measurement.  See https://www.youtube.com/watch?v=EMFCQB2qtc8.)

Internet2 is in the process of deploying 2x100G testers in all of its PoPs as part of NGI.  We did some back-to-back testing of single interfaces in our lab and got 94-ish Gb/s out of them with minimal effort.  Another thing we’ve been looking at is how Docker’s bridged macvlan to a host physical interface affects throughput, and it seems to be negligible.  We’ll share more about our experience once the project is fully-deployed.

--Mark