perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

On 05/12/2014 20:39, Brian Candler wrote:
> But: using bwctl I am still getting lower figures than iperf3 by 
> itself. What flags or options is bwctl running iperf3 with?

OK, I think I found it.

Short version: bwctl unconditionally sets the zerocopy (-Z) option on 
iperf3, and this is reducing the performance.

Long version: for iperf3 bwctl is *statically linked* against libiperf: 
it doesn't spawn the executable, but it calls iperf_set_XXX to set 
options, followed by iperf_run_server or iperf_run_client.

The difference between the iperf3 main() function and bwctl is that 
bwctl unconditionally sets some options in Iperf3PreRunTest:

    /* -t test duration in seconds */
    iperf_set_test_duration( iperf_test, tsess->test_spec.duration );

    /* -p server port */
    iperf_set_test_server_port( iperf_test, tsess->tool_port );

    /* -w window size in bytes */
    iperf_set_test_socket_bufsize( iperf_test, 
tsess->test_spec.window_size );

    // set zero-copy to get better performance
    iperf_set_test_zerocopy( iperf_test, 1 );

And hey presto: I find if I use the iperf3 command-line tool with the -Z 
option at the client side (it is rejected at the server side), I get the 
same degradation in performance:

bou-perf-1 ~]$ iperf3 -c cov-perf-1 -Z
Connecting to host cov-perf-1, port 5201
[  4] local 192.0.2.251 port 41733 connected to 192.0.2.254 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   469 MBytes  3.93 Gbits/sec    0    628 KBytes
[  4]   1.00-2.00   sec   948 MBytes  7.95 Gbits/sec    0    669 KBytes
[  4]   2.00-3.00   sec   866 MBytes  7.26 Gbits/sec    0    713 KBytes
[  4]   3.00-4.00   sec   880 MBytes  7.39 Gbits/sec    0    759 KBytes
[  4]   4.00-5.00   sec   954 MBytes  8.00 Gbits/sec    0    802 KBytes
[  4]   5.00-6.00   sec   760 MBytes  6.38 Gbits/sec    0    830 KBytes
[  4]   6.00-7.00   sec  1.00 GBytes  8.62 Gbits/sec    0    865 KBytes
[  4]   7.00-8.00   sec   779 MBytes  6.53 Gbits/sec    0    887 KBytes
[  4]   8.00-9.00   sec   944 MBytes  7.89 Gbits/sec    0    911 KBytes
[  4]   9.00-10.00  sec   812 MBytes  6.83 Gbits/sec    0    923 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.24 GBytes  7.08 Gbits/sec 0             sender
[  4]   0.00-10.00  sec  8.24 GBytes  7.08 Gbits/sec                  
receiver

iperf Done.

The underlying difference for the zerocopy option appears to be just 
this in iperf_tcp.c:

    if (sp->test->zerocopy)
        r = Nsendfile(sp->buffer_fd, sp->socket, sp->buffer, 
sp->settings->blksize);
    else
        r = Nwrite(sp->socket, sp->buffer, sp->settings->blksize, Ptcp);

Clearly, with these Intel X520-DA2 10G NICs, "write" is performing 
better than "sendfile" (although I don't fully understand the code; I 
can't see where the stream is being rewound for example)

In the general case, does -Z perform better for most people?

Should it just be turned off, or made a flag to bwctl?

Also note: the -Z flag doesn't make any difference if the client is 
initiating reverse mode (-R), which seems to be a limitation of the 
iperf code or protocol (it should pass this option over to the server end)

Regards,

Brian.