perfSONAR User Q&A and Other Discussion

[Brian Tierney <>]

Which version of OSX?

There was a new iperf3 release earlier this week that addressed some issues with Yosemite:

http://software.es.net/iperf/news.html#iperf-3-0-10-released

Maybe you could try compiling bwctl with this version, and see if thats any better.

Or maybe its a bug. ;)

On Thu, Dec 18, 2014 at 8:01 AM, Brian Candler <> wrote:

Here's an odd thing I see when running bwctl between a Linux box and an OSX box:

Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-10.00  sec  18.1 MBytes  15.2 Mbits/sec                  sender
[ 17]   0.00-10.00  sec  1.08 GBytes   926 Mbits/sec                  receiver

You can see that the values reported by the OSX end are ~60 times lower than the correct values reported by the Linux end (for the same stream/test!)  Full output below. It's the same with both IPv6 and IPv4.

Maybe some sort of integer overflow?

At the OSX end this is bwctl build from homebrew:
brew install bwctl --with-iperf3

Regards,

Brian Candler.


[example@cov-perf-1 ~]$ bwctl -v -x -s cov-client-1.network.example.com -T iperf3
Messages being sent to syslog(user,err)
bwctl: Using cov-client-1.network.example.com as the address for remote sender
bwctl: Using 2001:db8:0:2::254 as the address for local receiver
bwctl: Available in-common: iperf nuttcp iperf3
bwctl: Using tool: iperf3
bwctl: Server 'localhost' accepted test request at time 1418918004.512081
bwctl: Client 'cov-client-1.network.example.com' accepted test request at time 1418918004.512081
bwctl: 17 seconds until test results available

RECEIVER START
bwctl: start_endpoint: 3627906798.312744
bwctl: run_endpoint: receiver: 2001:db8:0:2::254
bwctl: run_endpoint: sender: 2001:db8:0:64::252
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 2001:db8:0:2::254
bwctl: run_tool: sender: 2001:db8:0:64::252
bwctl: start_tool: 3627906801.918425
iperf 3.0.9
Linux cov-perf-1.network.example.com 2.6.32-504.1.3.el6.aufs.web100.x86_64 #1 SMP Wed Nov 12 16:01:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------
Server listening on 5133
-----------------------------------------------------------
Time: Thu, 18 Dec 2014 15:53:24 GMT
Accepted connection from 2001:db8:0:64::252, port 49592
      Cookie: cov-client-1.network.example.com.1
      TCP MSS: 1428 (default)
[ 17] local 2001:db8:0:2::254 port 5133 connected to 2001:db8:0:64::252 port 49593
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-1.00   sec   110 MBytes   923 Mbits/sec
[ 17]   1.00-2.00   sec   110 MBytes   926 Mbits/sec
[ 17]   2.00-3.00   sec   110 MBytes   926 Mbits/sec
[ 17]   3.00-4.00   sec   110 MBytes   926 Mbits/sec
[ 17]   4.00-5.00   sec   110 MBytes   926 Mbits/sec
[ 17]   5.00-6.00   sec   110 MBytes   926 Mbits/sec
[ 17]   6.00-7.00   sec   110 MBytes   926 Mbits/sec
[ 17]   7.00-8.00   sec   110 MBytes   926 Mbits/sec
[ 17]   8.00-9.00   sec   110 MBytes   926 Mbits/sec
[ 17]   9.00-10.00  sec   110 MBytes   926 Mbits/sec
[ 17]  10.00-10.00  sec   169 KBytes   915 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-10.00  sec  18.1 MBytes  15.2 Mbits/sec                  sender
[ 17]   0.00-10.00  sec  1.08 GBytes   926 Mbits/sec                  receiver
CPU Utilization: local/receiver 20.4% (1.4%u/19.1%s), remote/sender 36.6% (2.7%u/34.0%s)
bwctl: stop_tool: 3627906814.910821
bwctl: stop_endpoint: 3627906815.912232

RECEIVER END

SENDER START
bwctl: start_endpoint: 3627906798.291852
bwctl: run_endpoint: receiver: 2001:db8:0:2::254
bwctl: run_endpoint: sender: 2001:db8:0:64::252
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 2001:db8:0:2::254
bwctl: run_tool: sender: 2001:db8:0:64::252
bwctl: start_tool: 3627906804.513476
iperf 3.0.9
Darwin cov-client-1.network.example.com 14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 x86_64
Time: Thu, 18 Dec 2014 15:53:24 GMT
Connecting to host 2001:db8:0:2::254, port 5133
      Cookie: cov-client-1.network.example.com.1
      TCP MSS: 1428 (default)
[ 17] local 2001:db8:0:64::252 port 49593 connected to 2001:db8:0:2::254 port 5133
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-1.00   sec  4.00 MBytes  33.6 Mbits/sec
[ 17]   1.00-2.00   sec  1.88 MBytes  15.7 Mbits/sec
[ 17]   2.00-3.00   sec  1.62 MBytes  13.6 Mbits/sec
[ 17]   3.00-4.00   sec  1.88 MBytes  15.7 Mbits/sec
[ 17]   4.00-5.00   sec  1.50 MBytes  12.6 Mbits/sec
[ 17]   5.00-6.00   sec  1.38 MBytes  11.5 Mbits/sec
[ 17]   6.00-7.00   sec  1.50 MBytes  12.6 Mbits/sec
[ 17]   7.00-8.00   sec  1.75 MBytes  14.7 Mbits/sec
[ 17]   8.00-9.00   sec  1.12 MBytes  9.44 Mbits/sec
[ 17]   9.00-10.00  sec  1.50 MBytes  12.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-10.00  sec  18.1 MBytes  15.2 Mbits/sec                  sender
[ 17]   0.00-10.00  sec  1.08 GBytes   926 Mbits/sec                  receiver
CPU Utilization: local/sender 36.6% (2.7%u/34.0%s), remote/receiver 20.4% (1.4%u/19.1%s)

iperf Done.
bwctl: stop_tool: 3627906814.889140
bwctl: stop_endpoint: 3627906815.889722

SENDER END
[example@cov-perf-1 ~]$ bwctl -v -x -s 192.0.2.252 -T iperf3
Messages being sent to syslog(user,err)
bwctl: Using 192.0.2.252 as the address for remote sender
bwctl: Using 192.0.2.254 as the address for local receiver
bwctl: Available in-common: iperf nuttcp iperf3
bwctl: Using tool: iperf3
bwctl: Server 'localhost' accepted test request at time 1418918085.798995
bwctl: Client '192.0.2.252' accepted test request at time 1418918085.798995
bwctl: 15 seconds until test results available

RECEIVER START
bwctl: start_endpoint: 3627906881.415730
bwctl: run_endpoint: receiver: 192.0.2.254
bwctl: run_endpoint: sender: 192.0.2.252
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 192.0.2.254
bwctl: run_tool: sender: 192.0.2.252
bwctl: start_tool: 3627906884.105830
iperf 3.0.9
Linux cov-perf-1.network.example.com 2.6.32-504.1.3.el6.aufs.web100.x86_64 #1 SMP Wed Nov 12 16:01:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------
Server listening on 5134
-----------------------------------------------------------
Time: Thu, 18 Dec 2014 15:54:45 GMT
Accepted connection from 192.0.2.252, port 49598
      Cookie: cov-client-1.network.example.com.1
      TCP MSS: 1448 (default)
[ 17] local 192.0.2.254 port 5134 connected to 192.0.2.252 port 49599
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-1.00   sec   112 MBytes   936 Mbits/sec
[ 17]   1.00-2.00   sec   112 MBytes   939 Mbits/sec
[ 17]   2.00-3.00   sec   112 MBytes   939 Mbits/sec
[ 17]   3.00-4.00   sec   112 MBytes   939 Mbits/sec
[ 17]   4.00-5.00   sec   112 MBytes   938 Mbits/sec
[ 17]   5.00-6.00   sec   112 MBytes   940 Mbits/sec
[ 17]   6.00-7.00   sec   112 MBytes   939 Mbits/sec
[ 17]   7.00-8.00   sec   112 MBytes   939 Mbits/sec
[ 17]   8.00-9.00   sec   112 MBytes   939 Mbits/sec
[ 17]   9.00-10.00  sec   112 MBytes   939 Mbits/sec
[ 17]  10.00-10.00  sec   153 KBytes   940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-10.00  sec  18.0 MBytes  15.1 Mbits/sec                  sender
[ 17]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver
CPU Utilization: local/receiver 22.1% (1.6%u/20.7%s), remote/sender 32.8% (2.3%u/30.5%s)
bwctl: stop_tool: 3627906895.911827
bwctl: stop_endpoint: 3627906896.912847

RECEIVER END

SENDER START
bwctl: start_endpoint: 3627906881.393818
bwctl: run_endpoint: receiver: 192.0.2.254
bwctl: run_endpoint: sender: 192.0.2.252
bwctl: run_tool: tester: iperf3
bwctl: run_tool: receiver: 192.0.2.254
bwctl: run_tool: sender: 192.0.2.252
bwctl: start_tool: 3627906885.800336
iperf 3.0.9
Darwin cov-client-1.network.example.com 14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 x86_64
Time: Thu, 18 Dec 2014 15:54:45 GMT
Connecting to host 192.0.2.254, port 5134
      Cookie: cov-client-1.network.example.com.1
      TCP MSS: 1448 (default)
[ 17] local 192.0.2.252 port 49599 connected to 192.0.2.254 port 5134
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-1.00   sec  4.38 MBytes  36.7 Mbits/sec
[ 17]   1.00-2.00   sec  1.38 MBytes  11.5 Mbits/sec
[ 17]   2.00-3.00   sec  1.38 MBytes  11.5 Mbits/sec
[ 17]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec
[ 17]   4.00-5.00   sec  1.75 MBytes  14.7 Mbits/sec
[ 17]   5.00-6.00   sec  1.62 MBytes  13.6 Mbits/sec
[ 17]   6.00-7.00   sec  1.62 MBytes  13.6 Mbits/sec
[ 17]   7.00-8.00   sec  1.62 MBytes  13.6 Mbits/sec
[ 17]   8.00-9.00   sec  1.62 MBytes  13.6 Mbits/sec
[ 17]   9.00-10.00  sec  1.38 MBytes  11.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[ 17]   0.00-10.00  sec  18.0 MBytes  15.1 Mbits/sec                  sender
[ 17]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver
CPU Utilization: local/sender 32.8% (2.3%u/30.5%s), remote/receiver 22.1% (1.6%u/20.7%s)

iperf Done.
bwctl: stop_tool: 3627906895.888968
bwctl: stop_endpoint: 3627906896.889973

SENDER END

--

Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net