perfSONAR User Q&A and Other Discussion

[Tim Chown <>]

Hi,

There’s a growing interest in 100G perfSONAR, so this is a very interesting 
topic.

Our experience is that iperf2 is more ‘friendly’ for higher throughput as it 
seems a little smarter on how it distributes the multiple streams to its 
processors, where iperf3 needs additional parameters to be set.  perfSONAR 
also supports Ethr for throughput testing as of 4.3.0, which we found to 
perform very well as an alternative to either iperf version, albeit using 
more CPU.

Some best practice guidance for 100G throughput tests would be useful.

Tim

> On 17 Jun 2021, at 21:59, Eli Dart <> wrote:
> 
> What happens if you run two streams?
> 
> It would be good to know if you're throughput-limited globally or 
> per-stream....
> 
> Thanks,
> 
> Eli
> 
> 
> 
> On Thu, Jun 17, 2021 at 9:07 AM "Fedorka, Shayne" 
> <> wrote:
> I have a new 100G perfSONAR deployment in early stages of testing and I am 
> consistently getting an average throughput between 30-40 Gbps. I’ve tried 
> various 100G tuning configurations (increased TCP buffer size, set CPU 
> governor to performance, updated NIC driver, etc.). I’m wondering if anyone 
> has any suggestions as to what else I should look at to get better 
> performance.
> 
>  
> 
> I have two servers connected to the same NVIDIA/Mellanox SN2010 switch on 
> the same LAN. The servers are identical with the following hardware:
> 
>       • Supermicro SuperServer
>       • 96GB Samsung Memory (6 x 16GB)
>       • NVIDIA/Mellanox ConnectX-5 Ethernet Card
>       • Intel 480GB SSD
>       • Intel Xeon Gold 3.6 GHz (4.4 GHz turbo) 8-core, 16-thread processor
>  
> 
> [perfsonar-100g-a ~]$ pscheduler task throughput -s 172.16.10.10 -d 
> 172.16.10.20 -b 100G
> 
> Submitting task...
> 
> Task URL:
> 
> https://172.16.10.10/pscheduler/tasks/4050bb29-b65a-476d-8d4d-6dd1b5c3668b
> 
> Running with tool 'iperf3'
> 
> Fetching first run...
> 
>  
> 
> Next scheduled run:
> 
> https://172.16.10.10/pscheduler/tasks/4050bb29-b65a-476d-8d4d-6dd1b5c3668b/runs/2739b7fa-03c3-4f23-860e-e8c3777f9c95
> 
> Starts 2021-06-17T14:57:26Z (~7 seconds)
> 
> Ends   2021-06-17T14:57:45Z (~18 seconds)
> 
> Waiting for result...
> 
>  
> 
> * Stream ID 5
> 
> Interval       Throughput     Retransmits    Current Window
> 
> 0.0 - 1.0      43.55 Gbps     0              15.88 MBytes  
> 
> 1.0 - 2.0      45.43 Gbps     0              19.59 MBytes  
> 
> 2.0 - 3.0      45.79 Gbps     0              19.59 MBytes  
> 
> 3.0 - 4.0      44.30 Gbps     0              30.67 MBytes  
> 
> 4.0 - 5.0      30.59 Gbps     0              30.67 MBytes  
> 
> 5.0 - 6.0      30.02 Gbps     0              30.67 MBytes  
> 
> 6.0 - 7.0      30.29 Gbps     0              30.67 MBytes  
> 
> 7.0 - 8.0      28.80 Gbps     0              30.67 MBytes  
> 
> 8.0 - 9.0      29.36 Gbps     0              30.67 MBytes  
> 
> 9.0 - 10.0     24.09 Gbps     0              30.67 MBytes  
> 
>  
> 
> Summary
> 
> Interval       Throughput     Retransmits    Receiver Throughput
> 
> 0.0 - 10.0     35.22 Gbps     0              35.01 Gbps
> 
>  
> 
> [perfsonar-100g-a ~]$ ifconfig
> 
> enp179s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
> 
>         inet 172.16.10.10  netmask 255.255.255.0  broadcast 172.16.10.255
> 
>         inet6 fe80::bace:f6ff:fe4e:c016  prefixlen 64  scopeid 0x20<link>
> 
>         ether b8:ce:f6:4e:c0:16  txqueuelen 10000  (Ethernet)
> 
>         RX packets 2500215  bytes 165614923 (157.9 MiB)
> 
>         RX errors 0  dropped 0  overruns 0  frame 0
> 
>         TX packets 34468724  bytes 309429388449 (288.1 GiB)
> 
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
>  
> 
> [perfsonar-100g-a ~]$ sudo ethtool enp179s0f0
> 
> Settings for enp179s0f0:
> 
>         Supported ports: [ FIBRE ]
> 
>         Supported link modes:   1000baseKX/Full
> 
>                                 10000baseKR/Full
> 
>                                 40000baseKR4/Full
> 
>                                 40000baseCR4/Full
> 
>                                 40000baseSR4/Full
> 
>                                 40000baseLR4/Full
> 
>                                 25000baseCR/Full
> 
>                                 25000baseKR/Full
> 
>                                 25000baseSR/Full
> 
>                                 50000baseCR2/Full
> 
>                                 50000baseKR2/Full
> 
>                                 100000baseKR4/Full
> 
>                                 100000baseSR4/Full
> 
>                                 100000baseCR4/Full
> 
>                                 100000baseLR4_ER4/Full
> 
>         Supported pause frame use: Symmetric
> 
>         Supports auto-negotiation: Yes
> 
>         Supported FEC modes: None RS
> 
>         Advertised link modes:  1000baseKX/Full
> 
>                                 10000baseKR/Full
> 
>                                 40000baseKR4/Full
> 
>                                 40000baseCR4/Full
> 
>                                 40000baseSR4/Full
> 
>                                 40000baseLR4/Full
> 
>                                 25000baseCR/Full
> 
>                                 25000baseKR/Full
> 
>                                 25000baseSR/Full
> 
>                                 50000baseCR2/Full
> 
>                                 50000baseKR2/Full
> 
>                                 100000baseKR4/Full
> 
>                                 100000baseSR4/Full
> 
>                                 100000baseCR4/Full
> 
>                                 100000baseLR4_ER4/Full
> 
>         Advertised pause frame use: Symmetric
> 
>         Advertised auto-negotiation: Yes
> 
>         Advertised FEC modes: RS
> 
>         Link partner advertised link modes:  Not reported
> 
>         Link partner advertised pause frame use: No
> 
>         Link partner advertised auto-negotiation: Yes
> 
>         Link partner advertised FEC modes: Not reported
> 
>         Speed: 100000Mb/s
> 
>         Duplex: Full
> 
>         Port: FIBRE
> 
>         PHYAD: 0
> 
>         Transceiver: internal
> 
>         Auto-negotiation: on
> 
>         Supports Wake-on: d
> 
>         Wake-on: d
> 
>         Current message level: 0x00000004 (4)
> 
>                                link
> 
>         Link detected: yes
> 
>  
> 
>  
> 
> -- 
> 
> Shayne Fedorka
> 
> Network Engineer | NREL
> 
> --
> To unsubscribe from this list: 
> https://lists.internet2.edu/sympa/signoff/perfsonar-user
> 
> 
> -- 
> 
> Eli Dart, Network Engineer                          NOC: (510) 486-7600
> ESnet Science Engagement Group                           (800) 333-7638
> Lawrence Berkeley National Laboratory 
> --
> To unsubscribe from this list: 
> https://lists.internet2.edu/sympa/signoff/perfsonar-user