perfSONAR User Q&A and Other Discussion

["Schopf, Jennifer M" <>]

If it’s helpful – EPOC’s view on 100G perfSONAR nodes (for your average campus)

https://epoc.global/wp-content/uploads/2020/12/2020-12-07-I-think-i-need-a-100G-perfSONAR-node-2.pdf

 

-j

 

------------------------------------------------

Dr. Jennifer M. Schopf

Director, International Networks

Director, Engagement and Performance Operations Center

Indiana University

 

 

From: <> on behalf of Mark Feit <>
Reply-To: Mark Feit <>
Date: Friday, June 18, 2021 at 10:40 AM
To: Tim Chown <>, Eli Dart <>
Cc: "Fedorka, Shayne" <>, "" <>
Subject: Re: [perfsonar-user] 100G perfSONAR Performance

 

Tim Chown writes:

 

Our experience is that iperf2 is more ‘friendly’ for higher throughput as it seems a little smarter on how it distributes the multiple streams to its processors, where iperf3 needs additional parameters to be set.

My understanding (and Bruce can correct me if I’m wrong) is that while it can be made to work for this use case, iperf3 wasn’t designed with it in mind.

 

perfSONAR also supports Ethr for throughput testing as of 4.3.0, which we found to perform very well as an alternative to either iperf version, albeit using more CPU.

If you can live without iperf3’s more-exotic features, iperf2 and Ethr  are the paths of least resistance for the time being.  Ethr’s only Achilles’ heel is that it was written in Go, whose garbage collection is tuned for latency at the expense of throughput.  This produces some observable blips in how much traffic it can make, but it was still capable of getting north of 90 Gb/s.  There is a new 1.x version of the program that we haven’t evaluated yet but will probably appear in the feature release of perfSONAR following 4.4.

If the current suite of tools in perfSONAR is inadequate, Ethr came about through a collaboration between the perfSONAR team and SDSC.  We’d be happy to work with anyone who’s found or developed something better and has a good understanding of how to use it.  This includes tools that do L2 measurements.

This is a good time to start thinking about how to measure at above 100 Gb/s.  Multi-400 Gb/s links are rapidly becoming a thing on backbones.

 

Some best practice guidance for 100G throughput tests would be useful.

Tuning the machine per the NIC vendor’s guidance should get you most of the way there.  In general, four cores seems to be the minimum number to get a good run in and you reach the point of diminishing returns at six.

One thing I can’t stress enough is that any evaluation of how these tools perform on your systems needs to be done in a lab setting on two systems with back-to-back interfaces.  Putting a network, especially a WAN, between them will get measurements of how the testers perform across the network, not how well the testers do.  (The first segment of my TechEXtra perfSONAR Day talk covers the principles of good measurement.  See https://www.youtube.com/watch?v=EMFCQB2qtc8.)

Internet2 is in the process of deploying 2x100G testers in all of its PoPs as part of NGI.  We did some back-to-back testing of single interfaces in our lab and got 94-ish Gb/s out of them with minimal effort.  Another thing we’ve been looking at is how Docker’s bridged macvlan to a host physical interface affects throughput, and it seems to be negligible.  We’ll share more about our experience once the project is fully-deployed.

--Mark

 

Attachment: smime.p7s
Description: S/MIME cryptographic signature