ndt-users list created

["Hao, Justin C" <>]

Howdy Rich,

So we have moved our 10Gig NDT servers into a production environment 
(connected via Cisco Nexus 5k and 7k switches) and we're still seeing the odd 
asynch results from ndt.  Server 1 is running ndt 3.6.3 and Server 2 is 
running ndt 3.6.4.  Each server has identical tcp configurations and 
hardware.  And as you can see in the attached results C2S is close to 10Gig 
as expected, but S2C in each case is not.  Let me know if you'd like me to 
test any changes on the servers to improve the ndt results or provide 
additional debugging information, thanks!

{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf320
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
\margl1440\margr1440\vieww13300\viewh15660\viewkind0
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural\pardirnatural

\f0\fs24 \cf0 web100clt -n x.x.x.31 -ll\
Testing network path for configuration and performance problems  --  Using IPv4 address\
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done\
checking for firewalls . . . . . . . . . . . . . . . . . . .  Done\
running 10s outbound test (client to server) . . . . .  9130.58 Mb/s\
running 10s inbound test (server to client) . . . . . . 1240.99 Mb/s\
Server unable to determine bottleneck link type.\
Information: Other network traffic is congesting the link\
Information [S2C]: Packet queuing detected: 84.70% (remote buffers)\
Client is probably behind a firewall. [Connection to the ephemeral port failed]\
\
	------  Web100 Detailed Analysis  ------\
\
Web100 reports the Round trip time = 1.59 msec;the Packet size = 8948 Bytes; and \
There were 25282 packets retransmitted, 102703 duplicate acks received, and 239998 SACK blocks received\
Packets arrived out-of-order 26.69% of the time.\
This connection is receiver limited 4.00% of the time.\
This connection is sender limited 72.86% of the time.\
This connection is network limited 23.14% of the time.\
\
    Web100 reports TCP negotiated the optional Performance Settings to: \
RFC 2018 Selective Acknowledgment: ON\
RFC 896 Nagle Algorithm: ON\
RFC 3168 Explicit Congestion Notification: OFF\
RFC 1323 Time Stamping: ON\
RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10\
The theoretical network limit is 2785.93 Mbps\
The NDT server has a 8192 KByte buffer which limits the throughput to 40276.90 Mbps\
Your PC/Workstation has a 6144 KByte buffer which limits the throughput to 30207.68 Mbps\
The network based flow control limits the throughput to 27496.16 Mbps\
\
Client Data reports link is ' -1', Client Acks report link is ' -1'\
Server Data reports link is ' -1', Server Acks report link is ' -1'\
Packet size is preserved End-to-End\
Server IP addresses are preserved End-to-End\
Client IP addresses are preserved End-to-End\
CurMSS: 8948\
X_Rcvbuf: 87380\
X_Sndbuf: 8388608\
AckPktsIn: 384808\
AckPktsOut: 0\
BytesRetrans: 226223336\
CongAvoid: 0\
CongestionOverCount: 39397\
CongestionSignals: 278\
CountRTT: 266657\
CurCwnd: 1628536\
CurRTO: 201\
CurRwinRcvd: 6078464\
CurRwinSent: 18432\
CurSsthresh: 2299636\
DSACKDups: 0\
DataBytesIn: 0\
DataBytesOut: 1804195536\
DataPktsIn: 0\
DataPktsOut: 1168772\
DupAcksIn: 102703\
ECNEnabled: 0\
FastRetran: 278\
MaxCwnd: 5726720\
MaxMSS: 8948\
MaxRTO: 207\
MaxRTT: 7\
MaxRwinRcvd: 6291456\
MaxRwinSent: 18432\
MaxSsthresh: 4295040\
MinMSS: 8948\
MinRTO: 201\
MinRTT: 0\
MinRwinRcvd: 18432\
MinRwinSent: 17896\
NagleEnabled: 1\
OtherReductions: 43334\
PktsIn: 384808\
PktsOut: 1168772\
PktsRetrans: 25282\
RcvWinScale: 10\
SACKEnabled: 3\
SACKsRcvd: 239998\
SendStall: 0\
SlowStart: 0\
SampleRTT: 1\
SmoothedRTT: 1\
SndWinScale: 10\
SndLimTimeRwin: 400605\
SndLimTimeCwnd: 2317287\
SndLimTimeSender: 7294874\
SndLimTransRwin: 4597\
SndLimTransCwnd: 2172\
SndLimTransSender: 6334\
SndLimBytesRwin: 496451852\
SndLimBytesCwnd: -1443205984\
SndLimBytesSender: -1544017628\
SubsequentTimeouts: 0\
SumRTT: 423684\
Timeouts: 0\
TimestampsEnabled: 1\
WinScaleRcvd: 10\
WinScaleSent: 10\
DupAcksOut: 0\
StartTimeUsec: 487918\
Duration: 10012771\
c2sData: -1\
c2sAck: -1\
s2cData: -1\
s2cAck: -1\
half_duplex: 0\
link: 100\
congestion: 1\
bad_cable: 0\
mismatch: 0\
spd: 1441.52\
bw: 2785.93\
loss: 0.000237856\
avgrtt: 1.59\
waitsec: 0.00\
timesec: 10.00\
order: 0.2669\
rwintime: 0.0400\
sendtime: 0.7286\
cwndtime: 0.2314\
rwin: 48.0000\
swin: 64.0000\
cwin: 43.6914\
rttsec: 0.001589\
Sndbuf: 8388608\
aspd: 0.00000\
CWND-Limited: 14191.00\
minCWNDpeak: 277388\
maxCWNDpeak: 4652960\
CWNDpeaks: 492\
}

{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf320
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
\margl1440\margr1440\vieww12760\viewh13460\viewkind0
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural\pardirnatural

\f0\fs24 \cf0 web100clt -n x.x.x.30 -ll\
Testing network path for configuration and performance problems  --  Using IPv4 address\
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done\
checking for firewalls . . . . . . . . . . . . . . . . . . .  Done\
running 10s outbound test (client to server) . . . . .  9430.54 Mb/s\
running 10s inbound test (server to client) . . . . . . 1118.64 Mb/s\
Server unable to determine bottleneck link type.\
Information: Other network traffic is congesting the link\
Information [S2C]: Packet queuing detected: 86.00% (remote buffers)\
Client is probably behind a firewall. [Connection to the ephemeral port failed]\
\
	------  Web100 Detailed Analysis  ------\
\
Web100 reports the Round trip time = 0.62 msec;the Packet size = 8948 Bytes; and \
There were 3889 packets retransmitted, 102089 duplicate acks received, and 253013 SACK blocks received\
Packets arrived out-of-order 25.56% of the time.\
This connection is receiver limited 17.72% of the time.\
This connection is sender limited 76.59% of the time.\
This connection is network limited 5.68% of the time.\
\
    Web100 reports TCP negotiated the optional Performance Settings to: \
RFC 2018 Selective Acknowledgment: ON\
RFC 896 Nagle Algorithm: ON\
RFC 3168 Explicit Congestion Notification: OFF\
RFC 1323 Time Stamping: ON\
RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10\
The theoretical network limit is 7316.76 Mbps\
The NDT server has a 6684 KByte buffer which limits the throughput to 84498.06 Mbps\
Your PC/Workstation has a 3956 KByte buffer which limits the throughput to 50010.03 Mbps\
The network based flow control limits the throughput to 39878.15 Mbps\
\
Client Data reports link is ' -1', Client Acks report link is ' -1'\
Server Data reports link is ' -1', Server Acks report link is ' -1'\
Packet size is preserved End-to-End\
Server IP addresses are preserved End-to-End\
Client IP addresses are preserved End-to-End\
CurMSS: 8948\
X_Rcvbuf: 87380\
X_Sndbuf: 6844560\
AckPktsIn: 399404\
AckPktsOut: 0\
BytesRetrans: 34798772\
CongAvoid: 0\
CongestionOverCount: 4754\
CongestionSignals: 258\
CountRTT: 263897\
CurCwnd: 17896\
CurRTO: 201\
CurRwinRcvd: 1068032\
CurRwinSent: 18432\
CurSsthresh: 993228\
DSACKDups: 0\
DataBytesIn: 0\
DataBytesOut: 1458563668\
DataPktsIn: 0\
DataPktsOut: 1131158\
DupAcksIn: 102089\
ECNEnabled: 0\
FastRetran: 258\
MaxCwnd: 3230228\
MaxMSS: 8948\
MaxRTO: 203\
MaxRTT: 3\
MaxRwinRcvd: 4050944\
MaxRwinSent: 18432\
MaxSsthresh: 1780652\
MinMSS: 8948\
MinRTO: 201\
MinRTT: 0\
MinRwinRcvd: 18432\
MinRwinSent: 17896\
NagleEnabled: 1\
OtherReductions: 25598\
PktsIn: 399404\
PktsOut: 1131158\
PktsRetrans: 3889\
RcvWinScale: 10\
SACKEnabled: 3\
SACKsRcvd: 253013\
SendStall: 0\
SlowStart: 0\
SampleRTT: 0\
SmoothedRTT: 1\
SndWinScale: 10\
SndLimTimeRwin: 1774337\
SndLimTimeCwnd: 569142\
SndLimTimeSender: 7668265\
SndLimTransRwin: 7233\
SndLimTransCwnd: 1745\
SndLimTransSender: 7929\
SndLimBytesRwin: 1616871292\
SndLimBytesCwnd: 700876176\
SndLimBytesSender: -859255640\
SubsequentTimeouts: 0\
SumRTT: 163036\
Timeouts: 0\
TimestampsEnabled: 1\
WinScaleRcvd: 10\
WinScaleSent: 10\
DupAcksOut: 0\
StartTimeUsec: 786221\
Duration: 10012611\
c2sData: -1\
c2sAck: -1\
s2cData: -1\
s2cAck: -1\
half_duplex: 0\
link: 100\
congestion: 1\
bad_cable: 0\
mismatch: 0\
spd: 1165.48\
bw: 7316.76\
loss: 0.000228085\
avgrtt: 0.62\
waitsec: 0.00\
timesec: 10.00\
order: 0.2556\
rwintime: 0.1772\
sendtime: 0.7659\
cwndtime: 0.0568\
rwin: 30.9062\
swin: 52.2198\
cwin: 24.6447\
rttsec: 0.000618\
Sndbuf: 6844560\
aspd: 0.00000\
CWND-Limited: 11955.00\
minCWNDpeak: 134220\
maxCWNDpeak: 2818620\
CWNDpeaks: 438\
}

-----
Justin Hao 
CCNA
Network Engineer, ITS Networking
The University of Texas at Austin

-----

On Sep 30, 2010, at 9:23 AM, Rich Carlson wrote:

> Hi Justin;
> 
> Please run the tests with the -ll option and post the results to this 
> list or just send them to me.  Off the top of my head I don't know why 
> the results would be different based on the direction.
> 
> Rich
> 
> On 9/29/2010 2:59 PM, Hao, Justin C wrote:
>> Howdy Rich,
>> 
>> I've got the boxes connected to each other, Server 1 is running ndt 3.6.3 
>> on CentOS 5.5 (2.6.35-web100 kernel) and Server 2 is running Perfsonar 
>> 3.2rc1 (looks like ndt 3.4.4a)
>> 
>> I'm still seeing the asynch C2S/S2C values and was wondering if you could 
>> shed any light/point me in the right direction.   I've included snapshots 
>> of the web100clt results for each server, please let me know if you need 
>> additional information. I've configured both servers identically in terms 
>> of sysctl.conf tcp settings as well as txqueuelen of 10000 and MTU of 9000
>> 
>> Server 1(10.0.0.1) to Server 2(10.0.0.2)
>> [root@dhcp-135-164
>>  etc]# web100clt -4 -n 10.0.0.2
>> Testing network path for configuration and performance problems  --  Using 
>> IPv4 address
>> Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
>> checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
>> running 10s outbound test (client to server) . . . . .  9876.02 Mb/s
>> running 10s inbound test (server to client) . . . . . . 3041.14 Mb/s
>> Server unable to determine bottleneck link type.
>> Information [S2C]: Packet queuing detected: 69.33% (local buffers)
>> Server '10.0.0.2' is not behind a firewall. [Connection to the ephemeral 
>> port was successful]
>> Client is not behind a firewall. [Connection to the ephemeral port was 
>> successful]
>> Packet size is preserved End-to-End
>> Server IP addresses are preserved End-to-End
>> Client IP addresses are preserved End-to-End
>> [root@dhcp-135-164
>>  etc]#
>> 
>> Server 2(10.0.0.2) to Server 1(10.0.0.1)
>> [root@localhost
>>  etc]# web100clt -4 -n 10.0.0.1
>> Testing network path for configuration and performance problems  --  Using 
>> IPv4 address
>> Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
>> checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
>> running 10s outbound test (client to server) . . . . .  9898.48 Mb/s
>> running 10s inbound test (server to client) . . . . . . 2622.73 Mb/s
>> Server unable to determine bottleneck link type.
>> Information: Other network traffic is congesting the link
>> Information [S2C]: Packet queuing detected: 72.39% (remote buffers)
>> Server '10.0.0.1' is not behind a firewall. [Connection to the ephemeral 
>> port was successful]
>> Client is not behind a firewall. [Connection to the ephemeral port was 
>> successful]
>> Packet size is preserved End-to-End
>> Server IP addresses are preserved End-to-End
>> Client IP addresses are preserved End-to-End
>> [root@localhost
>>  etc]#
>> 
>> 
>> -----
>> Justin Hao
>> CCNA
>> Network Engineer, ITS Networking
>> The University of Texas at Austin
>> 
>> -----
>> 
>> On Sep 29, 2010, at 9:01 AM, Hao, Justin C wrote:
>> 
>>> that was step two, i'm going to connect the two servers to each other and 
>>> see what i can see
>>> 
>>> -----
>>> Justin Hao
>>> CCNA
>>> Network Engineer, ITS Networking
>>> The University of Texas at Austin
>>> 
>>> -----
>>> 
>>> On Sep 29, 2010, at 9:00 AM, Rich Carlson wrote:
>>> 
>>>> Justin;
>>>> 
>>>> Use the latest JRE, but look at the console service.  On Win based
>>>> machines I've found the upgrade process doesn't remove old version of
>>>> the java console.  You need to manually do this through the control
>>>> panel.  I don't have a linux box handy to see what it does.
>>>> 
>>>> In any case, I'd say you aren't really testing the NDT tool on a
>>>> loopback interface, so running the server and client on a single machine
>>>> isn't really going to tell you much.  Bring up a server and use the
>>>> applet to test a number of different clients to see what is going on.
>>>> 
>>>> Rich
>>>> 
>>>> On 9/29/2010 9:37 AM, Hao, Justin C wrote:
>>>>> Howdy Rich,
>>>>> 
>>>>> I've tried with both the command line and the web client, only the web 
>>>>> client has returned the negative values, but the command line client 
>>>>> has also returned a wide range of results. what is the recommended java 
>>>>> RE to use?
>>>>> 
>>>>> -----
>>>>> Justin Hao
>>>>> CCNA
>>>>> Network Engineer, ITS Networking
>>>>> The University of Texas at Austin
>>>>> 
>>>>> -----
>>>>> 
>>>>> On Sep 29, 2010, at 8:30 AM, Rich Carlson wrote:
>>>>> 
>>>>>> Justin;
>>>>>> 
>>>>>> The Web100 system uses 32bit counters, so I suspect the negative speeds
>>>>>> come from the use of signed int vars instead of unsigned.
>>>>>> 
>>>>>> When testing on the local box, the traffic goes through the loopback
>>>>>> interface (lo0) instead of the NIC.  This means you are testing the OS
>>>>>> and it's memory management system more that anything else.
>>>>>> Changing/setting variables on the ethx interface will have no effect.
>>>>>> 
>>>>>> You should not set the tcp_mem value to 16 M.  The value for this
>>>>>> variable is in pages, NOT bytes.  See
>>>>>> Documentation/networking/ip-sysctl.txt doc in the kernel source tree 
>>>>>> for
>>>>>> more details.
>>>>>> 
>>>>>> tcp_mem - vector of 3 INTEGERs: min, pressure, max
>>>>>>        min: below this number of pages TCP is not bothered about its
>>>>>>        memory appetite.
>>>>>> 
>>>>>>        pressure: when amount of memory allocated by TCP exceeds this
>>>>>> number
>>>>>>        of pages, TCP moderates its memory consumption and enters memory
>>>>>>        pressure mode, which is exited when memory consumption falls
>>>>>>        under "min".
>>>>>> 
>>>>>>        max: number of pages allowed for queueing by all TCP sockets.
>>>>>> 
>>>>>>        Defaults are calculated at boot time from amount of available
>>>>>>        memory.
>>>>>> 
>>>>>> 
>>>>>> Are you using the command line client (web100clt) or the Java Applet
>>>>>> (via the browser) to run these tests?  It was recently reported that a
>>>>>> site was getting high variability in the S2C tests with the Java 
>>>>>> client.
>>>>>> It turned out that they had 2 Java consoles installed.  Removing the
>>>>>> older console cleared up the problem.
>>>>>> 
>>>>>> Regards;
>>>>>> Rich
>>>>>> 
>>>>>> On 9/27/2010 6:49 PM, Hao, Justin C wrote:
>>>>>>> So i'm setting up a pair of ndt test servers for our new datacenter 
>>>>>>> and running into some hurdles.
>>>>>>> 
>>>>>>> First off, i haven't connected them to each other (or to a 10Gig 
>>>>>>> network), so all my testing has been on a single box running loopback 
>>>>>>> tests to itself (i have no other 10GigE hosts available to me at the 
>>>>>>> moment).
>>>>>>> 
>>>>>>> I'm running CentOS 5.5 and patched kernel 2.6.35 with the proper 
>>>>>>> web100 version etc.
>>>>>>> 
>>>>>>> It's currently hooked up via 1GigE and i've run several different 
>>>>>>> clients against it and I get good results for 1Gig performance
>>>>>>> 
>>>>>>> I'm running loopback tests to tweak TCP settings while I wait for our 
>>>>>>> 10Gig environment to be made ready. I'm getting 15-18Gig/s for C2S 
>>>>>>> but for S2C i'm getting numbers all over the place. from 3.5Gig/s to 
>>>>>>> 500Mb/s.  and most particularly odd, i'm seeing negative numbers in 
>>>>>>> some of the test output.
>>>>>>> 
>>>>>>> I welcome any comments and suggestions for tuning this server (it's a 
>>>>>>> dell r610 w/ an intel 10GigE adapter)
>>>>>>> 
>>>>>>> note: I've also configured the ethernet interface to use 9000 MTU and 
>>>>>>> a txqueuelen of 10000
>>>>>>> 
>>>>>>> Server TCP settings:
>>>>>>> 
>>>>>>> # increase TCP max buffer size setable using setsockopt()
>>>>>>> 
>>>>>>> net.core.rmem_max = 16777216
>>>>>>> net.core.wmem_max = 16777216
>>>>>>> 
>>>>>>> # increase Linux autotuning TCP buffer limits
>>>>>>> # min, default, and max number of bytes to use
>>>>>>> # set max to 16MB for 1GE, and 32M or 54M for 10GE
>>>>>>> 
>>>>>>> net.ipv4.tcp_mem = 16777216 16777216 16777216
>>>>>>> net.ipv4.tcp_rmem = 10240 87380 16777216
>>>>>>> net.ipv4.tcp_wmem = 10240 65536 16777216
>>>>>>> 
>>>>>>> net.ipv4.tcp_window_scaling = 1
>>>>>>> 
>>>>>>> # don't cache ssthresh from previous connection
>>>>>>> net.ipv4.tcp_no_metrics_save = 1
>>>>>>> 
>>>>>>> # recommended to increase this for 10G NICS
>>>>>>> net.core.netdev_max_backlog = 262144
>>>>>>> 
>>>>>>> NDT Output:
>>>>>>> 
>>>>>>> TCP/Web100 Network Diagnostic Tool v3.6.3
>>>>>>> Click START to start the test
>>>>>>> 
>>>>>>> ** Starting test 1 of 1 **
>>>>>>> Connecting to '127.0.0.1' [/127.0.0.1] to run test
>>>>>>> Connected to: 127.0.0.1--  Using IPv4 address
>>>>>>> Checking for Middleboxes  . . . . . . . . . . . . . . . . . .  Done.
>>>>>>> Checking for firewalls . . . . . . . . . . . . . . . . . . .  Done.
>>>>>>> running 10s outbound test (client-to-server [C2S]) . . . . . 
>>>>>>> 13744.22Mb/s
>>>>>>> running 10s inbound test (server-to-client [S2C]) . . . . . . 
>>>>>>> -83702.57kb/s
>>>>>>> Server unable to determine bottleneck link type.
>>>>>>> [S2C]: Packet queueing detected
>>>>>>> 
>>>>>>> Click START to re-test
>>>>>>> 
>>>>>>> ** Starting test 1 of 1 **
>>>>>>> Connecting to '127.0.0.1' [/127.0.0.1] to run test
>>>>>>> Connected to: 127.0.0.1--  Using IPv4 address
>>>>>>> Checking for Middleboxes  . . . . . . . . . . . . . . . . . .  Done.
>>>>>>> Checking for firewalls . . . . . . . . . . . . . . . . . . .  Done.
>>>>>>> running 10s outbound test (client-to-server [C2S]) . . . . . 
>>>>>>> 12876.05Mb/s
>>>>>>> running 10s inbound test (server-to-client [S2C]) . . . . . . 
>>>>>>> 1006.86Mb/s
>>>>>>> Server unable to determine bottleneck link type.
>>>>>>> [S2C]: Packet queueing detected
>>>>>>> 
>>>>>>> Click START to re-test
>>>>>>> 
>>>>>>> ** Starting test 1 of 1 **
>>>>>>> Connecting to '127.0.0.1' [/127.0.0.1] to run test
>>>>>>> Connected to: 127.0.0.1--  Using IPv4 address
>>>>>>> Checking for Middleboxes  . . . . . . . . . . . . . . . . . .  Done.
>>>>>>> Checking for firewalls . . . . . . . . . . . . . . . . . . .  Done.
>>>>>>> running 10s outbound test (client-to-server [C2S]) . . . . . 
>>>>>>> 18466.0Mb/s
>>>>>>> running 10s inbound test (server-to-client [S2C]) . . . . . . 
>>>>>>> -1710004.63kb/s
>>>>>>> Server unable to determine bottleneck link type.
>>>>>>> [S2C]: Packet queueing detected
>>>>>>> 
>>>>>>> Click START to re-test
>>>>>>> 
>>>>>>> 
>>>>>>> -----
>>>>>>> Justin Hao
>>>>>>> CCNA
>>>>>>> Network Engineer, ITS Networking
>>>>>>> The University of Texas at Austin
>>>>>>> 
>>>>>>> -----
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
>>