ndt-users list created

[Richard Carlson <>]

Hi Dale;

I don't recall if I replied to this earlier but here it is again.

I'm seeing this problem more and more and I am currently trying to find a 
real fix for this problem.  Here's what's happening.

The NDT server starts sending data to the remote client.  It enters a 
simple send loop and pumps data into the network for 10 seconds.  It then 
exits the loop and sends the final results over to the client.

The problem is, that when in this loop it is possible for the OS to 
transfer data to the TCP stack, faster than the TCP stack pump it out into 
the network.  This results in a large standing queue, visible with the 
netstat -nat command).  So at the end of 10 seconds the code stops pumping 
more data but the client keeps reading until the queue empties.  Note the 
Web100 Duration variable has a value of 34,893,925 microseconds, or almost 
35 seconds.

One temporary step is to limit the max buffer space (tcp_wmax) to something 
in the 512 KB to 1MB range.  This will keep the queue from building up too 
much, but it's really just a band-aid until I can figure out how to monitor 
the queue length to prevent such large queues in the first place.

If anyone has any suggestions on how to do this, please let me know.

Rich

At 10:52 AM 2/10/2006, Dale Blount wrote:
> Hello again,
>
> I've just upgraded my old web100/ndt server to new hardware with onboard
> gigE (instead of a card) and a xeon instead of a p2.  The new server is
> also running 2.6.15 (the old one was running 2.6.11 or so).  It seems to
> run fine on the lan (my workstation only has 100mbps, and it clocks
> around 94mbps both ways).  However, any distant connection -
> DSL/Cable/dialup seems to test horribly in the downstream direction.
>
> I've tweaked the sysctl params the same they were on the other server,
> but they don't seem to help much.
>
> Downloading a file from the server appears to travel at normal speed.
>
> Below are the logs from a 5mb/5mb cable modem connection, 3 hops from
> the ndt server.
>
> Thanks for any pointers and/or tuning advice.
>
> Dale
>
>
>
>
>
>
>
> Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
> running 10s outbound test (client to server) . . . . . 3.73Mb/s
> running 10s inbound test (server to client) . . . . . . 72.66kb/s
> The slowest link in the end-to-end path is a 10 Mbps Ethernet subnet
> Information: The receive buffer should be 73.99 Kbytes to maximize
> throughput
>
> ********************************************************************
>
> WEB100 Enabled Statistics:
> Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
> running 10s outbound test (client to server) . . . . . 3.73Mb/s
> running 10s inbound test (server to client) . . . . . . 72.66kb/s
>
>         ------  Client System Details  ------
> OS data: Name = Windows 2003, Architecture = x86, Version = 5.2
> Java data: Vendor = Sun Microsystems Inc., Version = 1.5.0_04
>
>         ------  Web100 Detailed Analysis  ------
> 10 Mbps Ethernet link found.
> Link set to Full Duplex mode
> No network congestion discovered.
> Good network cable(s) found
> Normal duplex operation found.
>
> Web100 reports the Round trip time = 60.62 msec; the Packet size = 1460
> Bytes; and
> There were 201 packets retransmitted, 18 duplicate acks received, and 20
> SACK blocks received
> The connection stalled 1 times due to packet loss
> The connection was idle 0.26 seconds (0.76%) of the time
> This connection is network limited 99.64% of the time.
>
> Web100 reports TCP negotiated the optional Performance Settings to:
> RFC 2018 Selective Acknowledgment: ON
> RFC 896 Nagle Algorithm: ON
> RFC 3168 Explicit Congestion Notification: OFF
> RFC 1323 Time Stamping: OFF
> RFC 1323 Window Scaling: OFF
> Packet size is preserved End-to-End
> Server IP addresses are preserved End-to-End
> Information: Network Address Translation (NAT) box is modifying the
> Client's IP address
>         Server says [W.X.Y.Z] but Client says [Z.Y.X.W]
>
>
> *******************************************************************
>
> WEB100 Kernel Variables:
> Client: localhost/127.0.0.1
> AckPktsIn: 231
> AckPktsOut: 0
> BytesRetrans: 293460
> CongAvoid: 0
> CongestionOverCount: 0
> CongestionSignals: 2
> CountRTT: 13
> CurCwnd: 1460
> CurMSS: 1460
> CurRTO: 264
> CurRwinRcvd: 17520
> CurRwinSent: 5840
> CurSsthresh: 16060
> DSACKDups: 0
> DataBytesIn: 0
> DataBytesOut: 3403472
> DataPktsIn: 0
> DataPktsOut: 2248
> DupAcksIn: 18
> ECNEnabled: 0
> FastRetran: 1
> MaxCwnd: 21900
> MaxMSS: 1460
> MaxRTO: 264
> MaxRTT: 100
> MaxRwinRcvd: 17520
> MaxRwinSent: 5840
> MaxSsthresh: 146000
> MinMSS: 1460
> MinRTO: 224
> MinRTT: 24
> MinRwinRcvd: 17520
> MinRwinSent: 5840
> NagleEnabled: 1
> OtherReductions: 0
> PktsIn: 231
> PktsOut: 2248
> PktsRetrans: 201
> X_Rcvbuf: 4194304
> RcvWinScale: 2147483647
> SACKEnabled: 3
> SACKsRcvd: 20
> SendStall: 0
> SlowStart: 0
> SampleRTT: 100
> SmoothedRTT: 64
> X_Sndbuf: 4194304
> SndWinScale: 2147483647
> SndLimTimeRwin: 0
> SndLimTimeCwnd: 34764344
> SndLimTimeSender: 126942
> SndLimTransRwin: 0
> SndLimTransCwnd: 1
> SndLimTransSender: 1
> SndLimBytesRwin: 0
> SndLimBytesCwnd: 3403472
> SndLimBytesSender: 0
> SubsequentTimeouts: 0
> SumRTT: 788
> Timeouts: 1
> TimestampsEnabled: 0
> WinScaleRcvd: 2147483647
> WinScaleSent: 2147483647
> DupAcksOut: 0
> StartTimeUsec: 464790
> Duration: 34893925
> c2sData: 3
> c2sAck: 3
> s2cData: 1
> s2cAck: 1
> half_duplex: 0
> link: 100
> congestion: 0
> bad_cable: 0
> mismatch: 0
> spd: -1.78
> bw: 6.16
> loss: 0.000889680
> avgrtt: 60.62
> waitsec: 0.26
> timesec: 34.00
> order: 0.0779
> rwintime: 0.0000
> sendtime: 0.0036
> cwndtime: 0.9964
> rwin: 0.1337
> swin: 32.0000
> cwin: 0.1671
> rttsec: 0.060615
> Sndbuf: 4194304
> aspd: 0.19621
>
> Checking for mismatch on uplink
>         (speed > 50 [-1.78>50], (xmitspeed < 5) [3.73<5]
>         (rwintime > .9) [0>.9], (loss < .01) [8.89<.01]
> Checking for excessive errors condition
>         (loss/sec > .15) [2.61>.15], (cwndtime > .6) [0.99>.6],
>         (loss < .01) [8.89<.01], (MaxSsthresh > 0) [146000>0]
> Checking for 10 Mbps link
>         (speed < 9.5) [-1.78<9.5], (speed > 3.0) [-1.78>3.0]
>         (xmitspeed < 9.5) [3.73<9.5] (loss < .01) [8.89<.01], (mylink > 0)
> [10.0>0]
> Checking for Wireless link
>         (sendtime = 0) [0.00=0], (speed < 5) [-1.78<5]
>         (Estimate > 50 [6.16>50], (Rwintime > 90) [0>.90]
>         (RwinTrans/CwndTrans = 1) [0/1=1], (mylink > 0) [10.0>0]
> Checking for DSL/Cable Modem link
>         (speed < 2) [-1.78<2], (SndLimTransSender = 0) [1=0]
>         (SendTime = 0) [0.0036=0], (mylink > 0) [10.0>0]
> Checking for half-duplex condition
>         (rwintime > .95) [0>.95], (RwinTrans/sec > 30) [0>30],
>         (SenderTrans/sec > 30) [0.02>30], OR (mylink <= 10) [10.0<=10]
> Checking for congestion
>         (cwndtime > .02) [0.99>.02], (mismatch = 0) [0=0]
>         (MaxSsthresh > 0) [146000>0]
>
> estimate = 6.16 based on packet size = 11Kbits, RTT = 60.62msec, and
> loss = 8.8968E-4
> The theoretical network limit is 6.16 Mbps
> The NDT server has a 4096.0 KByte buffer which limits the throughput to
> 527.92 Mbps
> Your PC/Workstation has a 17.0 KByte buffer which limits the throughput
> to 2.20 Mbps
> The network based flow control limits the throughput to 2.75 Mbps
>
> Client Data reports link is 'Ethernet', Client Acks report link is
> 'Ethernet'
> Server Data reports link is 'Dial-up', Server Acks report link is
> 'Dial-up'

------------------------------------



Richard A. Carlson                              e-mail: 

Network Engineer                                phone:  (734) 352-7043
Internet2                                       fax:    (734) 913-4255
1000 Oakbrook Dr; Suite 300
Ann Arbor,  MI  48104