perfSONAR User Q&A and Other Discussion

[Brian Tierney <>]

That is a somewhat common pattern if there is a device in the path with buffers that are too small.

You can config perfsonar to use a smaller buffer using the GUI, or editing the mesh config file.

On Sat, Jun 18, 2016 at 6:42 PM, Eschen, Brian <> wrote:

Hello,

I have noticed an issue with my  long distance bandwidth tests, I believe that its mostly related to the TCP window size. For the last few months my tests have regularly been able to get between 7 and 8.5Gbps. Starting on June 11th around 3AM all of my tests are stuck at just under 2.4Gbps and have been ever since.  

I have been running some of my own bwctl iperf3 tests and almost all of them look the same, never going over 2.4Gbps.  If I set the –w  flag myself I can get test results that are back in the 7 – 8.5Gbps range. 

I have the sysctl.conf file configured with these settings:

# increase TCP max buffer size setable using setsockopt()
net.core.rmem_max = 536870912 
net.core.wmem_max = 536870912 
# increase Linux autotuning TCP buffer limit
net.ipv4.tcp_rmem = 4096 87380 268435456
net.ipv4.tcp_wmem = 4096 65536 268435456

# recommended to increase this for 10G NICS or higher

net.core.netdev_max_backlog = 250000

# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
# Explicitly set htcp as the congestion control: cubic buggy in older 2.6 kernels
net.ipv4.tcp_congestion_control=htcp
# If you are using Jumbo Frames, also set this
net.ipv4.tcp_mtu_probing = 1

SENDER START

Connecting to host 144.92.20.109, port 5626

[ 15] local 67.58.50.75 port 48525 connected to 144.92.20.109 port 5626

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[ 15]   0.00-1.00   sec   744 KBytes  6.09 Mbits/sec    0    236 KBytes       (omitted)

[ 15]   1.00-2.00   sec  9.27 MBytes  77.7 Mbits/sec    0   2.14 MBytes       (omitted)

[ 15]   2.00-3.00   sec   111 MBytes   933 Mbits/sec    0   21.8 MBytes       (omitted)

[ 15]   3.00-4.00   sec   276 MBytes  2317 Mbits/sec    0   21.8 MBytes       (omitted)

[ 15]   4.00-5.00   sec   255 MBytes  2139 Mbits/sec    0   21.8 MBytes       (omitted)

[ 15]   0.00-1.00   sec   276 MBytes  2317 Mbits/sec    0   21.8 MBytes

[ 15]   1.00-2.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes

[ 15]   2.00-3.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes

[ 15]   3.00-4.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes

[ 15]   4.00-5.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes

[ 15]   5.00-6.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes

[ 15]   6.00-7.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes

[ 15]   7.00-8.00   sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes

[ 15]   8.00-9.00   sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes

[ 15]   9.00-10.00  sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes

[ 15]  10.00-11.00  sec   275 MBytes  2307 Mbits/sec    0   21.8 MBytes

[ 15]  11.00-12.00  sec   256 MBytes  2150 Mbits/sec    0   21.8 MBytes

[ 15]  12.00-13.00  sec   274 MBytes  2296 Mbits/sec    0   21.8 MBytes

[ 15]  13.00-14.00  sec   258 MBytes  2160 Mbits/sec    0   21.8 MBytes

[ 15]  14.00-15.00  sec   274 MBytes  2295 Mbits/sec    0   21.8 MBytes

[ 15]  15.00-16.00  sec   258 MBytes  2161 Mbits/sec    0   21.8 MBytes

[ 15]  16.00-17.00  sec   272 MBytes  2286 Mbits/sec    0   21.8 MBytes

[ 15]  17.00-18.00  sec   259 MBytes  2171 Mbits/sec    0   21.8 MBytes

[ 15]  18.00-19.00  sec   272 MBytes  2286 Mbits/sec    0   21.8 MBytes

[ 15]  19.00-20.00  sec   258 MBytes  2160 Mbits/sec    0   21.8 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[ 15]   0.00-20.00  sec  5.19 GBytes  2228 Mbits/sec    0             sender

[ 15]   0.00-20.00  sec  5.21 GBytes  2237 Mbits/sec                  receiver

iperf Done.

Any ideas?

Thank you

—

Brian Eschen

Infrastructure Specialist

Morgridge Institute for Research 

--

Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net