perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

On 09/12/2014 16:34, Brian Candler wrote:
> > How long does it take before it spits back an error?
>
> I think it came back in a few seconds - quicker than the 15 seconds I 
> was expecting for a working test. Unfortunately, just to spite me, it 
> is working right now :-( 
I replicated it now: it comes back after 6.5 seconds (see below).

To replicate it I first tried stopping and restarting bwctl-server, but 
that didn't seem to make a difference. Then I uninstalled my 
bwctl-server rpm (with the no_zerocopy patch) on one node (bou-perf-1), 
and reinstalled the original rpm with yum; and that did the trick. In 
general it's been intermittent when uninstalling and installing 
packages. Smells like uninitialized data to me.

Note however: as it is right now, a test in the opposite direction 
(still initiated from bou-perf-1, but using -s cov-perf-1 instead of -c 
cov-perf-1) works.  cov-perf-1 is still running my patched rpm. The 
patch is only for no_zerocopy; I didn't modify anything else.

Regards,

Brian.


bou-perf-1 ~]$ time bwctl -a 2 -c cov-perf-1 -T iperf3
bwctl: NTP: Status UNSYNC (clock offset problems likely)
bwctl: Using tool: iperf3
bwctl: 15 seconds until test results available

SENDER START

SENDER END

real    0m6.526s
user    0m0.002s
sys    0m0.003s

bou-perf-1 ~]$ time bwctl -v -x -a 2 -c cov-perf-1 -T iperf3
Messages being sent to syslog(user,err)
bwctl: NTP: Status UNSYNC (clock offset problems likely)
bwctl: Using 192.0.2.251 as the address for local sender
bwctl: Using cov-perf-1 as the address for remote receiver
bwctl: Available in-common: iperf nuttcp iperf3
bwctl: Using tool: iperf3
bwctl: Server 'cov-perf-1' accepted test request at time 1418164188.777478
bwctl: Client 'localhost' accepted test request at time 1418164188.777478
bwctl: 15 seconds until test results available

RECEIVER START
bwctl: start_endpoint: 3627152984.376946
bwctl: run_endpoint: receiver: 192.0.2.254
bwctl: run_endpoint: sender: 192.0.2.251
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 192.0.2.254
bwctl: run_tool: sender: 192.0.2.251
bwctl: start_tool: 3627152987.076461
iperf 3.0.9
Linux cov-perf-1.network.example.com 
2.6.32-504.1.3.el6.aufs.web100.x86_64 #1 SMP Wed Nov 12 16:01:48 UTC 
2014 x86_64 x86_64 x86_64 GNU/Linux
iperf3: interrupt - the server has terminated
-----------------------------------------------------------
Server listening on 5602
-----------------------------------------------------------

bwctl: remote peer cancelled test
bwctl: stop_tool: 3627152990.804609
bwctl: stop_endpoint: 3627152990.805426

RECEIVER END

SENDER START
bwctl: start_endpoint: 3627152984.355103
bwctl: run_endpoint: receiver: 192.0.2.254
bwctl: run_endpoint: sender: 192.0.2.251
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 192.0.2.254
bwctl: run_tool: sender: 192.0.2.251
bwctl: start_tool: 3627152988.777648
iperf 3.0.9
Linux bou-perf-1.network.example.com 
2.6.32-504.1.3.el6.aufs.web100.x86_64 #1 SMP Wed Nov 12 16:01:48 UTC 
2014 x86_64 x86_64 x86_64 GNU/Linux
bwctl: stop_tool: 3627152988.781859
bwctl: stop_endpoint: 3627152990.783144

SENDER END

real    0m6.560s
user    0m0.002s
sys    0m0.003s


But:

bou-perf-1 ~]$ time bwctl -v -x -a 2 -s cov-perf-1 -T iperf3
Messages being sent to syslog(user,err)
bwctl: NTP: Status UNSYNC (clock offset problems likely)
bwctl: Using cov-perf-1 as the address for remote sender
bwctl: Using 192.0.2.251 as the address for local receiver
bwctl: Available in-common: iperf nuttcp iperf3
bwctl: Using tool: iperf3
bwctl: Server 'localhost' accepted test request at time 1418164453.536092
bwctl: Client 'cov-perf-1' accepted test request at time 1418164453.536092
bwctl: 15 seconds until test results available

RECEIVER START
bwctl: start_endpoint: 3627153249.135823
bwctl: run_endpoint: receiver: 192.0.2.251
bwctl: run_endpoint: sender: 192.0.2.254
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 192.0.2.251
bwctl: run_tool: sender: 192.0.2.254
bwctl: start_tool: 3627153251.835086
iperf 3.0.9
Linux bou-perf-1.network.example.com 
2.6.32-504.1.3.el6.aufs.web100.x86_64 #1 SMP Wed Nov 12 16:01:48 UTC 
2014 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------
Server listening on 5300
-----------------------------------------------------------
Time: Tue, 09 Dec 2014 22:34:13 GMT
Accepted connection from 192.0.2.254, port 45472
      Cookie: cov-perf-1.network.example.com.141
      TCP MSS: 1448 (default)
[ 17] local 192.0.2.251 port 5300 connected to 192.0.2.254 port 39488
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 
seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-1.00   sec   694 MBytes  5.83 Gbits/sec
[ 17]   1.00-2.00   sec  1022 MBytes  8.57 Gbits/sec
[ 17]   2.00-3.00   sec  1.01 GBytes  8.67 Gbits/sec
[ 17]   3.00-4.00   sec  1.01 GBytes  8.67 Gbits/sec
[ 17]   4.00-5.00   sec  1.01 GBytes  8.65 Gbits/sec
[ 17]   5.00-6.00   sec  1.01 GBytes  8.66 Gbits/sec
[ 17]   6.00-7.00   sec  1.01 GBytes  8.66 Gbits/sec
[ 17]   7.00-8.00   sec  1.01 GBytes  8.65 Gbits/sec
[ 17]   8.00-9.00   sec  1.01 GBytes  8.65 Gbits/sec
[ 17]   9.00-10.00  sec  1.01 GBytes  8.63 Gbits/sec
[ 17]  10.00-10.04  sec  36.9 MBytes  8.59 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[ 17]   0.00-10.04  sec  9.78 GBytes  8.37 Gbits/sec 0             sender
[ 17]   0.00-10.04  sec  9.77 GBytes  8.37 Gbits/sec                  
receiver
CPU Utilization: local/receiver 84.3% (0.4%u/84.0%s), remote/sender 
69.9% (0.4%u/69.6%s)
bwctl: stop_tool: 3627153263.679135
bwctl: stop_endpoint: 3627153264.679855

RECEIVER END

SENDER START
bwctl: start_endpoint: 3627153249.158695
bwctl: run_endpoint: receiver: 192.0.2.251
bwctl: run_endpoint: sender: 192.0.2.254
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 192.0.2.251
bwctl: run_tool: sender: 192.0.2.254
bwctl: start_tool: 3627153253.536244
iperf 3.0.9
Linux cov-perf-1.network.example.com 
2.6.32-504.1.3.el6.aufs.web100.x86_64 #1 SMP Wed Nov 12 16:01:48 UTC 
2014 x86_64 x86_64 x86_64 GNU/Linux
Time: Tue, 09 Dec 2014 22:34:13 GMT
Connecting to host 192.0.2.251, port 5300
      Cookie: cov-perf-1.network.example.com.141
      TCP MSS: 1448 (default)
[ 16] local 192.0.2.254 port 39488 connected to 192.0.2.251 port 5300
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 
seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[ 16]   0.00-1.00   sec   728 MBytes  6.10 Gbits/sec    0    568 KBytes
[ 16]   1.00-2.00   sec  1.01 GBytes  8.64 Gbits/sec    0    650 KBytes
[ 16]   2.00-3.00   sec  1.01 GBytes  8.68 Gbits/sec    0    697 KBytes
[ 16]   3.00-4.00   sec  1.01 GBytes  8.67 Gbits/sec    0    762 KBytes
[ 16]   4.00-5.00   sec  1.01 GBytes  8.65 Gbits/sec    0    807 KBytes
[ 16]   5.00-6.00   sec  1.01 GBytes  8.66 Gbits/sec    0    844 KBytes
[ 16]   6.00-7.00   sec  1.01 GBytes  8.66 Gbits/sec    0    875 KBytes
[ 16]   7.00-8.00   sec  1.01 GBytes  8.65 Gbits/sec    0    898 KBytes
[ 16]   8.00-9.00   sec  1.01 GBytes  8.65 Gbits/sec    0    923 KBytes
[ 16]   9.00-10.00  sec  1.01 GBytes  8.64 Gbits/sec    0    938 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[ 16]   0.00-10.00  sec  9.78 GBytes  8.40 Gbits/sec 0             sender
[ 16]   0.00-10.00  sec  9.77 GBytes  8.39 Gbits/sec                  
receiver
CPU Utilization: local/sender 69.9% (0.4%u/69.6%s), remote/receiver 
84.3% (0.4%u/84.0%s)

iperf Done.
bwctl: stop_tool: 3627153263.701509
bwctl: stop_endpoint: 3627153264.702199

SENDER END

real    0m15.648s
user    0m0.002s
sys    0m0.005s