perfSONAR User Q&A and Other Discussion

[Aaron Brown <>]

Hey Brian,

Could you try running with the “-v” and “-x” flags?

Cheers,
Aaron

> On Dec 4, 2014, at 10:07 AM, Brian Candler 
> <>
>  wrote:
> 
> I have two perfsonar 3.4.1 boxes set up next to each other. They are HP 
> Proliant 726042-425 with quad-core E3-1220 v3 @ 3.1GHz, 8GB RAM, 10G Intel 
> X520-DA2 NIC connected back-to-back with an SFP+ direct attach cable.
> 
> For reference below, the hostnames are "cov-perf-1" and "pb-perf-1".
> 
> If I run iperf3 directly from the command line, I get good and consistent 
> results:
> 
> [on cov-perf-1] iperf3 -s
> 
> [on pb-perf-1] $ iperf3 -c cov-perf-1
> Connecting to host cov-perf-1, port 5201
> [  4] local 192.0.2.125 port 52124 connected to 192.0.2.254 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec   787 MBytes  6.60 Gbits/sec    0    587 KBytes
> [  4]   1.00-2.00   sec  1.01 GBytes  8.64 Gbits/sec    0    675 KBytes
> [  4]   2.00-3.00   sec  1.01 GBytes  8.66 Gbits/sec    0    741 KBytes
> [  4]   3.00-4.00   sec  1.01 GBytes  8.65 Gbits/sec    0    776 KBytes
> [  4]   4.00-5.00   sec  1.01 GBytes  8.65 Gbits/sec    0    827 KBytes
> [  4]   5.00-6.00   sec  1.01 GBytes  8.65 Gbits/sec    0    865 KBytes
> [  4]   6.00-7.00   sec  1.01 GBytes  8.65 Gbits/sec    0    904 KBytes
> [  4]   7.00-8.00   sec  1.01 GBytes  8.66 Gbits/sec    0    928 KBytes
> [  4]   8.00-9.00   sec  1.01 GBytes  8.66 Gbits/sec    0    970 KBytes
> [  4]   9.00-10.00  sec  1.01 GBytes  8.65 Gbits/sec    0    980 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-10.00  sec  9.83 GBytes  8.45 Gbits/sec 0             sender
> [  4]   0.00-10.00  sec  9.83 GBytes  8.44 Gbits/sec                  
> receiver
> 
> iperf Done.
> 
> However if I run apparently the "same" test with bwctl I get lower and more 
> variable results:
> 
> [pb-perf-1] $ bwctl -s cov-perf-1 -T iperf3
> bwctl: Using tool: iperf3
> bwctl: 19 seconds until test results available
> 
> SENDER START
> Connecting to host 192.0.2.125, port 5126
> [ 16] local 192.0.2.254 port 56340 connected to 192.0.2.125 port 5126
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [ 16]   0.00-1.00   sec   694 MBytes  5.82 Gbits/sec    0    532 KBytes
> [ 16]   1.00-2.00   sec   821 MBytes  6.89 Gbits/sec    0    576 KBytes
> [ 16]   2.00-3.00   sec   759 MBytes  6.36 Gbits/sec    0    614 KBytes
> [ 16]   3.00-4.00   sec   830 MBytes  6.96 Gbits/sec    0    655 KBytes
> [ 16]   4.00-5.00   sec   710 MBytes  5.96 Gbits/sec    0    721 KBytes
> [ 16]   5.00-6.00   sec   811 MBytes  6.80 Gbits/sec    0    754 KBytes
> [ 16]   6.00-7.00   sec   955 MBytes  8.00 Gbits/sec    0    799 KBytes
> [ 16]   7.00-8.00   sec   827 MBytes  6.94 Gbits/sec    0    813 KBytes
> [ 16]   8.00-9.00   sec   712 MBytes  5.97 Gbits/sec    0    837 KBytes
> [ 16]   9.00-10.00  sec   811 MBytes  6.81 Gbits/sec    0    846 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [ 16]   0.00-10.00  sec  7.74 GBytes  6.65 Gbits/sec 0             sender
> [ 16]   0.00-10.00  sec  7.74 GBytes  6.65 Gbits/sec                  
> receiver
> 
> iperf Done.
> 
> SENDER END
> 
> Furthermore, if I try to run it in the opposite direction, I get no results 
> at all:
> 
> [pb-perf-1] $ bwctl -c cov-perf-1 -T iperf3
> bwctl: Using tool: iperf3
> bwctl: 19 seconds until test results available
> 
> SENDER START
> 
> SENDER END
> 
> $ bwctl -s pb-perf-1 -c cov-perf-1 -T iperf3
> bwctl: Using tool: iperf3
> bwctl: 19 seconds until test results available
> 
> SENDER START
> 
> SENDER END
> 
> ps shows that three new bwctld processes/threads appear during the test (at 
> both sides), but no iperf/iperf3 processes.
> 
> If I use strace -f -p <pid-of-bwctld> I do not see any fork or exec, but 
> two instances of clone(), so maybe these are threads running directly 
> within the bwctld daemon?
> 
> "top" shows the process running at nice 0, so that's not the problem.
> 
> Can anyone explain why these figures are worse with bwctl than native 
> iperf3, and why the tests don't work in both directions?
> 
> Thanks,
> 
> Brian Candler.
> 
> P.S. I have the default unmodified limits file in place, which has 
> "bandwidth=900m". Even assuming this means 900 Megabits per second rather 
> than 900 millibits per second, clearly I'm achieving more than that (or is 
> this supposed to be a long-term average from a particular IP address, 
> rather than an instantaneous peak?)
> 
> The file also contains this comment:
> 
> #       This example configuration allows Abilene measurement hosts to run
> #       tests to your server, but will not allow you to run tests to your 
> own
> #       hosts. This is intentional so you will have to look at this file and
> #       modifiy it.
> 
> But I can see nothing specific to Abilene.