ndt-users list created

[Richard Carlson <>]

Hi Maurice;

Thanks for the feedback.  I am currently in the process of setting up 
a XEN based server so I will experiment with this in the next few 
weeks.  I suspect that the problem is related to the NIC packet 
processing.  The time stamps used for the link estimation algorithm 
are added when the kernel processes the packets (libpcap 
processing).  This is a known problem when the local NIC handles some 
of the off-load functions or does interrupt coalescing.

Right now I don't have a solution, but I'd suggest the following:

*  Turn off TCP Segmentation Offload (TSO) using "ethtool -K eth0 tso off"
*  Set the NIC to interrupt after every packet using "ethtool -C eth0 
rx-frames 1"

These will force the kernel to handle all the traffic, which is what 
should happen for this type of tester.

You can also send me part of the web100srv.log file.  The "spds[?]" 
lines show the link detection timings.  It would be good to see what 
they look like.  Finally you can tell the NDT server to save some 
TCPDUMP files by adding the '-t' flag to the web100srv process.  This 
would allow you to look at the packet arrival times to see what 
libpcap was doing.

Rich

At 07:34 PM 10/2/2007, Maurice Volaski wrote:
> Since web100 kernel patches aren't stable enough to run on a system 
> doing other stuff, I've put together a (Gentoo) Linux guest OS on 
> VMWare, which is hosted by (Gentoo) Linux.
>
> The underlying NICs are motherboard-attached Broadcom 5704 and uses 
> the tg3 driver and it's attached to a Cisco 6509 at gigabit. The 
> guest OS is in its own VLAN on the switch.
>
> I've been having weird results with my Mac sometimes testing as if 
> it had a duplex mismatch. I don't know what's causing that. But when 
> it does work (another odd thing is that it sometimes works, 
> sometimes not), NDT insists I have just a 10 Mb connection, even 
> though it's gigabit and it's testing at half its full throughput. 
> Could this have something to do with the vmxnet driver that handles 
> the Ethernet in guest OS?
>
>
>
> WEB100 Enabled Statistics:
> Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
> checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
> running 10s outbound test (client-to-server [C2S]) . . . . . 470.75Mb/s
> running 10s inbound test (server-to-client [S2C]) . . . . . . 402.88Mb/s
>
>         ------  Client System Details  ------
> OS data: Name = Mac OS X, Architecture = i386, Version = 10.4.10
> Java data: Vendor = Apple Computer, Inc., Version = 1.5.0_07
>
>         ------  Web100 Detailed Analysis  ------
> 10 Mbps Ethernet link found.
> Link set to Full Duplex mode
> No network congestion discovered.
> Good network cable(s) found
> Normal duplex operation found.
>
> Web100 reports the Round trip time = 0.52 msec; the Packet size = 
> 1448 Bytes; and
> No packet loss - but packets arrived out-of-order 1.32% of the time
> C2S throughput test: Packet queuing detected: 0.01%
> S2C throughput test: Packet queuing detected: 1.06%
> This connection is sender limited 99.37% of the time.
>
> Web100 reports TCP negotiated the optional Performance Settings to:
> RFC 2018 Selective Acknowledgment: ON
> RFC 896 Nagle Algorithm: ON
> RFC 3168 Explicit Congestion Notification: OFF
> RFC 1323 Time Stamping: ON
> RFC 1323 Window Scaling: OFF
>
> Server 'gordonmacone.aecom.yu.edu' is not behind a firewall. 
> [Connection to the ephemeral port was successful]
> Client is not behind a firewall. [Connection to the ephemeral port 
> was successful]
> Information: Network Middlebox is modifying MSS variable
> Server IP addresses are preserved End-to-End
> Client IP addresses are preserved End-to-End
> WEB100 Kernel Variables:
> Client: localhost/127.0.0.1
> X_Rcvbuf: 212992
> X_Sndbuf: 212992
> AckPktsIn: 74991
> AckPktsOut: 0
> BytesRetrans: 0
> CongAvoid: 0
> CongestionOverCount: 0
> CongestionSignals: 0
> CountRTT: 71950
> CurCwnd: 112944
> CurMSS: 1448
> CurRTO: 204
> CurRwinRcvd: 65535
> CurRwinSent: 5792
> CurSsthresh: 2147483647
> DSACKDups: 0
> DataBytesIn: 0
> DataBytesOut: 520605184
> DataPktsIn: 0
> DataPktsOut: 359792
> DupAcksIn: 987
> ECNEnabled: 0
> FastRetran: 0
> MaxCwnd: 112944
> MaxMSS: 1448
> MaxRTO: 204
> MaxRTT: 8
> MaxRwinRcvd: 65535
> MaxRwinSent: 5792
> MaxSsthresh: 0
> MinMSS: 1448
> MinRTO: 204
> MinRTT: 0
> MinRwinRcvd: 44432
> MinRwinSent: 5792
> NagleEnabled: 1
> OtherReductions: 0
> PktsIn: 74991
> PktsOut: 359792
> PktsRetrans: 0
> RcvWinScale: 4
> SACKEnabled: 3
> SACKsRcvd: 0
> SendStall: 0
> SlowStart: 0
> SampleRTT: 0
> SmoothedRTT: 4
> SndWinScale: 0
> SndLimTimeRwin: 52007
> SndLimTimeCwnd: 12002
> SndLimTimeSender: 10025141
> SndLimTransRwin: 409
> SndLimTransCwnd: 44
> SndLimTransSender: 437
> SndLimBytesRwin: 4414840
> SndLimBytesCwnd: 667480
> SndLimBytesSender: 515522864
> SubsequentTimeouts: 0
> SumRTT: 37460
> Timeouts: 0
> TimestampsEnabled: 1
> WinScaleRcvd: 0
> WinScaleSent: 4
> DupAcksOut: 0
> StartTimeUsec: 695584
> Duration: 10089150
> c2sData: 3
> c2sAck: 6
> s2cData: 3
> s2cAck: 4
> half_duplex: 0
> link: 100
> congestion: 0
> bad_cable: 0
> mismatch: 0
> spd: 412.80
> bw: 21218.84
> loss: 0.000001000
> avgrtt: 0.52
> waitsec: 0.00
> timesec: 10.00
> order: 0.0132
> rwintime: 0.0052
> sendtime: 0.9937
> cwndtime: 0.0012
> rwin: 0.5000
> swin: 1.6250
> cwin: 0.8617
> rttsec: 0.000521
> Sndbuf: 212992
> aspd: 0.00000
> CWND-Limited: 184.18
>
> The theoretical network limit is 21218.84 Mbps
> The NDT server has a 104.0 KByte buffer which limits the throughput 
> to 3119.00 Mbps
> Your PC/Workstation has a 63.0 KByte buffer which limits the 
> throughput to 959.69 Mbps
> The network based flow control limits the throughput to 1653.93 Mbps
>
> Client Data reports link is 'Ethernet', Client Acks report link is 'OC-12'
> Server Data reports link is 'Ethernet', Server Acks report link is 'T3'
> --
>
> Maurice Volaski, 
>
> Computing Support, Rose F. Kennedy Center
> Albert Einstein College of Medicine of Yeshiva University

------------------------------------



Richard A. Carlson                              e-mail: 

Network Engineer                                phone:  (734) 352-7043
Internet2                                       fax:    (734) 913-4255
1000 Oakbrook Dr; Suite 300
Ann Arbor,  MI  48104