perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hey Jared;

TSO/LRO basically tries to unburden the CPU by making the NIC work a little harder (e.g. the CPU passes larger data chunks, the NIC splits them into network friendly bites).  Usually a good idea to have these for LAN based things, sometimes causes problems on the WAN though. 

Not all NICs let you mess with the settings, some do via ethtool:

    https://fasterdata.es.net/host-tuning/nic-tuning/

E.g. "ethtool -K $INTERFACE tso off"

Turning off/on and running some quick tests will let you know - its more art than science unfortunately. 

Thanks;

-jason

Jared Schlemmer wrote:

Thank you Brent. The multiples of 32 is a good observation. I’m not terrible familiar with TSO/LRO and its impact - is this something that can be tuned to lessen the impact, real or perceived, in the tests? 

Jared Schlemmer

Network Engineer, GlobalNOC at Indiana University

On Jul 31, 2017, at 4:25 PM, Brent Draney <> wrote:

I notice that you are loosing a multiple of 32.  This suggests that some combination of TSO/LRO may be at play

and the test may be getting perturbed by the nic offload masking some of the real counter drops.  You may have

a smaller drop rate than this counter suggests.

Thanks,

Brent

On Jul 31, 2017, at 10:52 AM, Eli Dart <> wrote:

​Hi Jared,

My first guess is that the latency between Monterey and Sunnyvale is low enough that the test is able to recover even with that packet loss.  I'll bet that a lot of the traffic that is seeing performance impact is traveling further.

Can you try a test from Monterey to Chicago or Monterey to New York? (or to someplace else more than 20 milliseconds away?)

Thanks,

Eli

​

On Mon, Jul 31, 2017 at 10:40 AM, Jared Schlemmer <> wrote:

We just turned up a new network endpoint that connects to an existing aggregation site via a 1gb AT&T VPLS connection and I’m seeing some interesting performance results. The sites are Monterey Bay and Sunnyvale, CA. Tests from Sunnyvale to Monterey Bay are good, but the reverse direction, Monterey Bay toward Sunnyvale, I see this:

Connecting to host port 5332
[ 16] local port 58534 connected to port 5332
[ ID] Interval                 Transfer        Bitrate              Retr  Cwnd
[ 16]   0.00-1.00   sec   110 MBytes  0.92 Gbits/sec    0   1.16 MBytes
[ 16]   1.00-2.00   sec   113 MBytes  0.95 Gbits/sec   64    553 KBytes
[ 16]   2.00-3.00   sec   111 MBytes  0.93 Gbits/sec   32    498 KBytes
[ 16]   3.00-4.00   sec   112 MBytes  0.94 Gbits/sec   32    434 KBytes
[ 16]   4.00-5.00   sec   112 MBytes  0.94 Gbits/sec   32    362 KBytes
[ 16]   5.00-6.00   sec   112 MBytes  0.94 Gbits/sec    0    669 KBytes
[ 16]   6.00-7.00   sec   112 MBytes  0.94 Gbits/sec   32    622 KBytes
[ 16]   7.00-8.00   sec   111 MBytes  0.93 Gbits/sec   32    574 KBytes
[ 16]   8.00-9.00   sec   112 MBytes  0.94 Gbits/sec   32    519 KBytes
[ 16]   9.00-10.00  sec   112 MBytes  0.94 Gbits/sec   32    458 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[ 16]   0.00-10.00  sec  1.09 GBytes  0.94 Gbits/sec  288             sender
[ 16]   0.00-10.00  sec  1.09 GBytes  0.93 Gbits/sec                  receiver

My questions are, a) how is it that we see retries and such a small window size and yet still get near line-rate throughput, and b) what is the real world impact of a test like this? Users at the Monterey site are reporting wildly varying performance out to the internet.

There are likely a lot of factors going on here, but I wanted to focus just on the testing between these two sites through the AT&T cloud. Any insights, theories or suggestions would be much appreciated. Thanks,


Jared Schlemmer
Network Engineer, GlobalNOC at Indiana University

--

Eli Dart, Network Engineer                          NOC: (510) 486-7600

ESnet Science Engagement Group                           (800) 333-7638

Lawrence Berkeley National Laboratory