ndt-users list created

[Clarke Morledge <>]

Rich,

I am looking for some clarification here:  it makes sense to say that the 
packet reordering is happening on the client-to-server (C2S) side since 
that would explain why the throughput is significantly lower in that 
direction when compared to the server-to-client (S2C) test.

But when you say that the "No packet loss - but packets arrived 
out-of-order" message indicates that the *client* is sending lots of dup 
ACKs, I got a little confused.  There should be very little TCP data 
running from the server to the client during the C2S throughput test, so I 
would think that the packet re-ordering in that direction would be less 
likely.  My guess would have been that the *server* would be more likely 
to generate dup ACKs since the client is the one generating the bulk of 
the packets.  Can you please correct me if I am wrong on this?

As far as everything else you asked, yes, I am testing in a gigabit LAN 
environment with an "out-of-the-box" client, so no window scaling and no 
TCP buffer size tweaking is going on.  The NDT server is on one layer 3 
subnet and the client is on an adjacent layer 3 subnet -- the rest of the 
path is all layer 2.  Moving the client around to different ports on the 
same switch and using different patch cables does not make any difference 
in the results.  However, if I move the client to a different layer 3
subnet, my results may differ and get more symmetric (what I would 
expect).

What is perplexing to me is if the data path on the network should be 
linear -- mostly layer 2 with but one layer 3 hop, what would be the 
potential sources for introducing the packet reordering?  Now, the server 
that I am using to run NDT is using the latest Network Performance Tools 
Knoppix CD on a system with dual processors.  Is it possible that the 
kernel is doing some mulithreading in such a way that it could potentially 
introduce some packet reordering?  I have my doubts about that, but 
perhaps someone else who knows more about the web100 kernel can help me 
out here.

Thanks for the feedback.

Clarke Morledge
College of William and Mary
Information Technology - Network Engineering
Jones Hall (Room 18)
Williamsburg VA 23187

On Thu, 13 Dec 2007, Richard Carlson wrote:

> Hi Clarke;
>
> All the detailed data reported by the NDT tool comes from the 
> Server-to-Client test.  This is because the web100 kernel captures the most 
> details while sending.  Since the server has the web100 enhanced kernel, the 
> most details come when the server is sending data to the client.
>
> The message below indicates that the client is generating a large percentage 
> of duplicate ACKs.  Duplicate ACKs occur when the client receives a packet 
> out of order.  If the packet was really lost, the server would also record a 
> large number of fast retransmits.  If the packets were just reordered, then 
> the number of fast retransmits should be low.
>
> To diagnose this problem, it would really help to set the expectation level 
> correctly.  Based on the fact that the S2C test is over 100 Mbps, can I 
> safely assume that this is a gigabit Ethernet network?  The RTT value says 
> that this is a LAN envrionment, but there is not a large queue beiing built 
> up on the server.  The client is using a 64 KB buffer and window scaling is 
> probably turned off.  Based on the RTT and TCP buffer size this connection 
> should max out around 135 Mbps, which is what you are getting.
>
> One think you can try is to start the web100srv process with the "-r" option. 
> This will cause it to log the web100 variable for the C2S test.  There might 
> be something in there that can help you gain a little more understanding. 
> Another thing to try is add the "-t" option to the web100srv process.  This 
> will cause the server to generate tcpdump files for both s2c and c2s tests. 
> These files can be analyzed to determine what is going on at the packet 
> level.
>
> Finally, what kind of infrastructure is between the client and server?  How 
> many switches and/or routers?  Have you tired moving the client to another 
> switch port?  How about trying another Ethernet drop cable?
>
> Rich
> At 05:33 PM 12/13/2007, Clarke Morledge wrote:
> > I am trying to interpret some of the output from NDT (v5.4.8) that 
> > indicates assymetrical performance variance.   The key component for me is 
> > the line below:
> >
> > "No packet loss - but packets arrived out-of-order 12.58% of the time"
> >
> > But the problem I have is that it isn't clear to me where the packets are 
> > arriving out-of-order.  In this case I am assuming that it is from the 
> > client-to-server direction, but could there also be some out-of-order in 
> > the server-to-client direction, too?  The output isn't clear.
> >
> > However, is the packets arriving out-of-order component really helping me 
> > to diagnose the assymetry problem, or is this a red herring?  Is there 
> > some relevant web-100 variable(s) that I should be looking at instead?
> >
> > Thanks.
> > -----------------------
> >
> > WEB100 Enabled Statistics:
> > Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
> > checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
> > running 10s outbound test (client-to-server [C2S]) . . . . . 40.08Mb/s
> > running 10s inbound test (server-to-client [S2C]) . . . . . . 134.32Mb/s
> >
> >    ------  Client System Details  ------
> > OS data: Name = Windows XP, Architecture = x86, Version = 5.1
> > Java data: Vendor = Sun Microsystems Inc., Version = 1.6.0_02
> >
> >    ------  Web100 Detailed Analysis  ------
> > 622 Mbps OC-12 link found.
> > Link set to Full Duplex mode
> > No network congestion discovered.
> > Good network cable(s) found
> > Normal duplex operation found.
> >
> > Web100 reports the Round trip time = 3.27 msec; the Packet size = 1460 
> > Bytes; and
> > No packet loss - but packets arrived out-of-order 12.58% of the time
> > C2S throughput test: Packet queuing detected: 0.00%
> > S2C throughput test: Excessive packet queuing detected: 10.87%
> > This connection is receiver limited 58.26% of the time.
> >  Increasing the the client's receive buffer (63.0 KB) will improve 
> > performance
> > This connection is sender limited 41.27% of the time.