perfSONAR User Q&A and Other Discussion

[Michael Smitasin <>]

This may not be the problem you're running into, but I noticed a similar 
issue... starting at different times, 3 of our "regular" perfSonar nodes 
show drops in regular throughput tests to various "far away" ESnet 
perfSonar nodes... 30-Mar, 5-May, and 25-May. Two of the three are 
connected to our border routers (Juniper MX480/960), and the third is 
connected to our ScienceDMZ router (Juniper MX480) with no topology 
changes that would correspond to these dates, so I would expect them to 
all have good buffers and clean paths.

Comparing /etc/sysctl.conf between my dev box and the "regular" 
perfSonar nodes, I noticed a couple extra config stanzas appended to the 
end, with the last having lower values. Example of just the 
net.core.wmem_max value:

> # cat sysctl.conf | grep core\.wmem | grep -v \#
> net.core.wmem_max = 268435456
> net.core.wmem_max = 268435456
> net.core.wmem_max = 67108864

I deleted the last stanza, restarted network and performed the test 
again and now I'm getting the throughput I expect. Unfortunately, I 
don't yet have config management set up on these hosts to see if the 
addition of the extra stanzas corresponds to the dates I saw performance 
dips.

Michael Smitasin
Network Engineer
LBLnet Services Group
Lawrence Berkeley National Laboratory


On 6/18/2016 9:42 AM, Eschen, Brian wrote:
> Hello,
>
> I have noticed an issue with my  long distance bandwidth tests, I believe 
> that its mostly related to the TCP window size. For the last few months my 
> tests have regularly been able to get between 7 and 8.5Gbps. Starting on June 
> 11th around 3AM all of my tests are stuck at just under 2.4Gbps and have been 
> ever since.
>
> I have been running some of my own bwctl iperf3 tests and almost all of them 
> look the same, never going over 2.4Gbps.  If I set the –w  flag myself I can 
> get test results that are back in the 7 – 8.5Gbps range.
>
> I have the sysctl.conf file configured with these settings:
>
> # increase TCP max buffer size setable using setsockopt()
> net.core.rmem_max = 536870912
> net.core.wmem_max = 536870912
> # increase Linux autotuning TCP buffer limit
> net.ipv4.tcp_rmem = 4096 87380 268435456
> net.ipv4.tcp_wmem = 4096 65536 268435456
>
> # recommended to increase this for 10G NICS or higher
>
> net.core.netdev_max_backlog = 250000
>
> # don't cache ssthresh from previous connection
> net.ipv4.tcp_no_metrics_save = 1
> # Explicitly set htcp as the congestion control: cubic buggy in older 2.6 
> kernels
> net.ipv4.tcp_congestion_control=htcp
> # If you are using Jumbo 
> Frames<http://fasterdata.es.net/network-tuning/mtu-issues/>, also set this
> net.ipv4.tcp_mtu_probing = 1
>
> SENDER START
> Connecting to host 144.92.20.109, port 5626
> [ 15] local 67.58.50.75 port 48525 connected to 144.92.20.109 port 5626
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [ 15]   0.00-1.00   sec   744 KBytes  6.09 Mbits/sec    0    236 KBytes       
> (omitted)
> [ 15]   1.00-2.00   sec  9.27 MBytes  77.7 Mbits/sec    0   2.14 MBytes       
> (omitted)
> [ 15]   2.00-3.00   sec   111 MBytes   933 Mbits/sec    0   21.8 MBytes       
> (omitted)
> [ 15]   3.00-4.00   sec   276 MBytes  2317 Mbits/sec    0   21.8 MBytes       
> (omitted)
> [ 15]   4.00-5.00   sec   255 MBytes  2139 Mbits/sec    0   21.8 MBytes       
> (omitted)
> [ 15]   0.00-1.00   sec   276 MBytes  2317 Mbits/sec    0   21.8 MBytes
> [ 15]   1.00-2.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes
> [ 15]   2.00-3.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes
> [ 15]   3.00-4.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes
> [ 15]   4.00-5.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes
> [ 15]   5.00-6.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes
> [ 15]   6.00-7.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes
> [ 15]   7.00-8.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes
> [ 15]   8.00-9.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes
> [ 15]   9.00-10.00  sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes
> [ 15]  10.00-11.00  sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes
> [ 15]  11.00-12.00  sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes
> [ 15]  12.00-13.00  sec   274 MBytes  2296 Mbits/sec    0   21.8 MBytes
> [ 15]  13.00-14.00  sec   258 MBytes  2160 Mbits/sec    0   21.8 MBytes
> [ 15]  14.00-15.00  sec   274 MBytes  2295 Mbits/sec    0   21.8 MBytes
> [ 15]  15.00-16.00  sec   258 MBytes  2161 Mbits/sec    0   21.8 MBytes
> [ 15]  16.00-17.00  sec   272 MBytes  2286 Mbits/sec    0   21.8 MBytes
> [ 15]  17.00-18.00  sec   259 MBytes  2171 Mbits/sec    0   21.8 MBytes
> [ 15]  18.00-19.00  sec   272 MBytes  2286 Mbits/sec    0   21.8 MBytes
> [ 15]  19.00-20.00  sec   258 MBytes  2160 Mbits/sec    0   21.8 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [ 15]   0.00-20.00  sec  5.19 GBytes  2228 Mbits/sec    0             sender
> [ 15]   0.00-20.00  sec  5.21 GBytes  2237 Mbits/sec                  receiver
>
> iperf Done.
>
>
> Any ideas?
> Thank you
> —
> Brian Eschen
> Infrastructure Specialist
> Morgridge Institute for Research