perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

I have two perfsonar 3.4.1 boxes set up next to each other. They are HP 
Proliant 726042-425 with quad-core E3-1220 v3 @ 3.1GHz, 8GB RAM, 10G 
Intel X520-DA2 NIC connected back-to-back with an SFP+ direct attach cable.

For reference below, the hostnames are "cov-perf-1" and "pb-perf-1".

If I run iperf3 directly from the command line, I get good and 
consistent results:

[on cov-perf-1] iperf3 -s

[on pb-perf-1] $ iperf3 -c cov-perf-1
Connecting to host cov-perf-1, port 5201
[  4] local 192.0.2.125 port 52124 connected to 192.0.2.254 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   787 MBytes  6.60 Gbits/sec    0    587 KBytes
[  4]   1.00-2.00   sec  1.01 GBytes  8.64 Gbits/sec    0    675 KBytes
[  4]   2.00-3.00   sec  1.01 GBytes  8.66 Gbits/sec    0    741 KBytes
[  4]   3.00-4.00   sec  1.01 GBytes  8.65 Gbits/sec    0    776 KBytes
[  4]   4.00-5.00   sec  1.01 GBytes  8.65 Gbits/sec    0    827 KBytes
[  4]   5.00-6.00   sec  1.01 GBytes  8.65 Gbits/sec    0    865 KBytes
[  4]   6.00-7.00   sec  1.01 GBytes  8.65 Gbits/sec    0    904 KBytes
[  4]   7.00-8.00   sec  1.01 GBytes  8.66 Gbits/sec    0    928 KBytes
[  4]   8.00-9.00   sec  1.01 GBytes  8.66 Gbits/sec    0    970 KBytes
[  4]   9.00-10.00  sec  1.01 GBytes  8.65 Gbits/sec    0    980 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.83 GBytes  8.45 Gbits/sec 0             sender
[  4]   0.00-10.00  sec  9.83 GBytes  8.44 Gbits/sec                  
receiver

iperf Done.

However if I run apparently the "same" test with bwctl I get lower and 
more variable results:

[pb-perf-1] $ bwctl -s cov-perf-1 -T iperf3
bwctl: Using tool: iperf3
bwctl: 19 seconds until test results available

SENDER START
Connecting to host 192.0.2.125, port 5126
[ 16] local 192.0.2.254 port 56340 connected to 192.0.2.125 port 5126
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[ 16]   0.00-1.00   sec   694 MBytes  5.82 Gbits/sec    0    532 KBytes
[ 16]   1.00-2.00   sec   821 MBytes  6.89 Gbits/sec    0    576 KBytes
[ 16]   2.00-3.00   sec   759 MBytes  6.36 Gbits/sec    0    614 KBytes
[ 16]   3.00-4.00   sec   830 MBytes  6.96 Gbits/sec    0    655 KBytes
[ 16]   4.00-5.00   sec   710 MBytes  5.96 Gbits/sec    0    721 KBytes
[ 16]   5.00-6.00   sec   811 MBytes  6.80 Gbits/sec    0    754 KBytes
[ 16]   6.00-7.00   sec   955 MBytes  8.00 Gbits/sec    0    799 KBytes
[ 16]   7.00-8.00   sec   827 MBytes  6.94 Gbits/sec    0    813 KBytes
[ 16]   8.00-9.00   sec   712 MBytes  5.97 Gbits/sec    0    837 KBytes
[ 16]   9.00-10.00  sec   811 MBytes  6.81 Gbits/sec    0    846 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[ 16]   0.00-10.00  sec  7.74 GBytes  6.65 Gbits/sec 0             sender
[ 16]   0.00-10.00  sec  7.74 GBytes  6.65 Gbits/sec                  
receiver

iperf Done.

SENDER END

Furthermore, if I try to run it in the opposite direction, I get no 
results at all:

[pb-perf-1] $ bwctl -c cov-perf-1 -T iperf3
bwctl: Using tool: iperf3
bwctl: 19 seconds until test results available

SENDER START

SENDER END

$ bwctl -s pb-perf-1 -c cov-perf-1 -T iperf3
bwctl: Using tool: iperf3
bwctl: 19 seconds until test results available

SENDER START

SENDER END

ps shows that three new bwctld processes/threads appear during the test 
(at both sides), but no iperf/iperf3 processes.

If I use strace -f -p <pid-of-bwctld> I do not see any fork or exec, but 
two instances of clone(), so maybe these are threads running directly 
within the bwctld daemon?

"top" shows the process running at nice 0, so that's not the problem.

Can anyone explain why these figures are worse with bwctl than native 
iperf3, and why the tests don't work in both directions?

Thanks,

Brian Candler.

P.S. I have the default unmodified limits file in place, which has 
"bandwidth=900m". Even assuming this means 900 Megabits per second 
rather than 900 millibits per second, clearly I'm achieving more than 
that (or is this supposed to be a long-term average from a particular IP 
address, rather than an instantaneous peak?)

The file also contains this comment:

#       This example configuration allows Abilene measurement hosts to run
#       tests to your server, but will not allow you to run tests to 
your own
#       hosts. This is intentional so you will have to look at this file and
#       modifiy it.

But I can see nothing specific to Abilene.