perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Greetings Jared;

Clarification questions:

    - Are both hosts connected at 1G NIC-to-switch bottlenecks, or are 
they connected at 10G yet have a 1G step-down in the path to traverse 
the circuit?

    - Is the latency between Sunnvale and Monterey about 5ms give or take?

    - 'whats in the path' besides the AT&T gear? Can you describe the 
things you know are on the edges?

To your questions,

    a) 1G over ~5ms doesn't need much window - a bit more than 512K 
will do.

    b) TCP is designed to back off.  If your testing was competing with 
local needs, it would result in lower performance.  Regarding what 
user's are seeing 'it depends'.  Are they all trying to run bwctl, or 
are enterprise things 'slow'.  Get more data on what they are unhappy 
about first if possible.  My guess is that a commodity circuit of that 
type has oversubscription somewhere in the path - and that will lead to 
variable performance.

Thanks;

-jason

Jared Schlemmer wrote:
> We just turned up a new network endpoint that connects to an existing aggregation 
> site via a 1gb AT&T VPLS connection and I’m seeing some interesting 
> performance results. The sites are Monterey Bay and Sunnyvale, CA. Tests from 
> Sunnyvale to Monterey Bay are good, but the reverse direction, Monterey Bay 
> toward Sunnyvale, I see this:
>
> Connecting to host port 5332
> [ 16] local port 58534 connected to port 5332
> [ ID] Interval                 Transfer        Bitrate              Retr  Cwnd
> [ 16]   0.00-1.00   sec   110 MBytes  0.92 Gbits/sec    0   1.16 MBytes
> [ 16]   1.00-2.00   sec   113 MBytes  0.95 Gbits/sec   64    553 KBytes
> [ 16]   2.00-3.00   sec   111 MBytes  0.93 Gbits/sec   32    498 KBytes
> [ 16]   3.00-4.00   sec   112 MBytes  0.94 Gbits/sec   32    434 KBytes
> [ 16]   4.00-5.00   sec   112 MBytes  0.94 Gbits/sec   32    362 KBytes
> [ 16]   5.00-6.00   sec   112 MBytes  0.94 Gbits/sec    0    669 KBytes
> [ 16]   6.00-7.00   sec   112 MBytes  0.94 Gbits/sec   32    622 KBytes
> [ 16]   7.00-8.00   sec   111 MBytes  0.93 Gbits/sec   32    574 KBytes
> [ 16]   8.00-9.00   sec   112 MBytes  0.94 Gbits/sec   32    519 KBytes
> [ 16]   9.00-10.00  sec   112 MBytes  0.94 Gbits/sec   32    458 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [ 16]   0.00-10.00  sec  1.09 GBytes  0.94 Gbits/sec  288             sender
> [ 16]   0.00-10.00  sec  1.09 GBytes  0.93 Gbits/sec                  receiver
>
> My questions are, a) how is it that we see retries and such a small window 
> size and yet still get near line-rate throughput, and b) what is the real 
> world impact of a test like this? Users at the Monterey site are reporting 
> wildly varying performance out to the internet.
>
> There are likely a lot of factors going on here, but I wanted to focus just on 
> the testing between these two sites through the AT&T cloud. Any insights, 
> theories or suggestions would be much appreciated. Thanks,
>
>
> Jared Schlemmer
> Network Engineer, GlobalNOC at Indiana University