ndt-users list created

[Richard Carlson <>]

Hi Clarke;

At 02:57 PM 12/17/2007, Clarke Morledge wrote:
> Rich,
>
> I am looking for some clarification here:  it makes sense to say 
> that the packet reordering is happening on the client-to-server 
> (C2S) side since that would explain why the throughput is 
> significantly lower in that direction when compared to the 
> server-to-client (S2C) test.

OK, I'll try and provide that clarification.  TCP performance depends 
on a large number of factors.   An incomplete list includes (1) TCP 
buffer settings on both hosts; (2) NIC card vendor and drivers; (3) 
Host activity and memory utilization; (4) Network activity; and (5) 
Application behavior.  Both the client and server are involved in 
this communication process and faults or issues with either host, or 
the network infrastructure, will result is lower than expected throughput.

Given the complex nature of this problem, the NDT system can help 
highlight some of the possible problem areas.  Unfortunately the NDT 
system may not always report everything, and this is why we are 
continuing to upgrade the system.  In some cases, more in-depth 
analysis and data collection techniques are required to augment the 
data the NDT server collects.

> But when you say that the "No packet loss - but packets arrived 
> out-of-order" message indicates that the *client* is sending lots of 
> dup ACKs, I got a little confused.  There should be very little TCP 
> data running from the server to the client during the C2S throughput 
> test, so I would think that the packet re-ordering in that direction 
> would be less likely.  My guess would have been that the *server* 
> would be more likely to generate dup ACKs since the client is the 
> one generating the bulk of the packets.  Can you please correct me 
> if I am wrong on this?

As I said in my previous email, the detailed web100 data variables 
listed under the "More Details" page, is generated during the S2C 
test.  This data is then analyzed by the NDT server to generate the 
diagnostic messages.  So the "No packet loss ..." message indicates 
that the client was returning duplicate ACK packets about 12.6% of the time.

The question is, why would the client be generating these Duplicate 
ACK packets?  There are 3 possibilities.  (1) the server is sending 
packets out of order.  Since this is a dual processor machine, this 
is possible - but unlikely.  The Linux kernel does have the ability 
to bind an IRQ to a single processor, and this is turned on in the 
2.6.20.10 kernel being used by the V1.8 NPToolkit disk.  (2) the 
network infrastructure can re-order packets.  This use to occur when 
multiple links are bonded together to form a logical higher capacity 
link.  Packets were placed into the various link queues in a 
round-robin approach, and this could cause reordering when other 
traffic was using the link.  Most vendors changed from a round-robin 
queuing algorithm to a hash based queuing algorithm to ensure that 
all packets use the same link.  (3) the receiver can cause reordering 
as well.  Packets are received by the NIC and placed in onboard 
memory while the host interrupt is pending.  Once the interrupt is 
processed the packets are moved across the I/O bus and then processed 
by the TCP stack.  Packets may be reordered if the NIC driver or OS 
doesn't handle the interrupts properly.  My first guess would be that 
the receiver is causing these mis-ordering events.  To verify this, 
it would be necessary to run a packet sniffer on the NDT server, the 
client, and a 3rd party host listening on the clients port (mirror 
the data to another port).

> As far as everything else you asked, yes, I am testing in a gigabit 
> LAN environment with an "out-of-the-box" client, so no window 
> scaling and no TCP buffer size tweaking is going on.  The NDT server 
> is on one layer 3 subnet and the client is on an adjacent layer 3 
> subnet -- the rest of the path is all layer 2.  Moving the client 
> around to different ports on the same switch and using different 
> patch cables does not make any difference in the results.  However, 
> if I move the client to a different layer 3
> subnet, my results may differ and get more symmetric (what I would expect).

So this means that there must be a router in the path.  If moving the 
client to another IP subnet causes the client to change behavior, 
then I would also want to know what the router is doing.  I assume 
that each IP subnet is connected to a different router interface.

> What is perplexing to me is if the data path on the network should 
> be linear -- mostly layer 2 with but one layer 3 hop, what would be 
> the potential sources for introducing the packet reordering?  Now, 
> the server that I am using to run NDT is using the latest Network 
> Performance Tools Knoppix CD on a system with dual processors.  Is 
> it possible that the kernel is doing some mulithreading in such a 
> way that it could potentially introduce some packet reordering?  I 
> have my doubts about that, but perhaps someone else who knows more 
> about the web100 kernel can help me out here.

Here are some steps that I would take to gain more details about what 
is going on.

1) NDT server tasks
        a) run 'top -d2' and monitor the output.  This will report 
server activity
            along with CPU utilization every 2 seconds.  Is anything CPU or
            memory bound?
        b) turn on more NDT server side logging.  Add the -r, -t, 
and --snaplog options
            to the web100srv process.  This will cause the system to 
write detailed
            TCPDUMP and SNAPLOG files for each test.  The -r flag 
will cause the
            server to log C2S details as well.

2) Network infrastructure
        a) watch the router CPU and memory stats during a test.  Look to see
            if the router is being hammered buy a test
        b) look at the router interface statistics.  Are there any 
errors or suspious
            log messages coming off the router.

3) Client side tasks
        a) run the Win-XP task manager and look at the performance statistics
            while a test is running.  Specifically look at CPU, 
memory, and network
            utilization numbers.
        b) create a TCPDUMP file using the WINDUMP command.  See
            http://www.winpcap.org/windump/ for details.  Once you 
have these DUMP
            files, the TCPTRACE command 
http://jarok.cs.ohiou.edu/software/tcptrace/
            will give you more details about what went on.

        c) If possible you can also create a mirrored port anywhere 
in this path and
            use another packet capture host to see what is really 
going on over the
            wire and what is being done inside each hosts (TCPDUMP 
only records
            packets after they have been received by the kernel so 
any NIC processing
            will be hidden.

        d) another possible action is to boot this client using 
another copy of the
            NPToolkit disk.  After it boots, login and enter the 
'startx' command.  This
            will bring up the xwindows system and you can run an NDT 
test using the
            same HW but a different OS/NIC driver.

        e) A 3rd possible solution is to try an NPAD test.  The 
NPToolkit disk contains
            another diagnostic tool being developed by the 
researchers at PCS.  Simply
            point your browser to port 8200 (instead of 7123) and 
follow the instructions.

That should keep you busy for awhile .-)

Rich

> Thanks for the feedback.
>
> Clarke Morledge
> College of William and Mary
> Information Technology - Network Engineering
> Jones Hall (Room 18)
> Williamsburg VA 23187
>
> On Thu, 13 Dec 2007, Richard Carlson wrote:
>
> > Hi Clarke;
> >
> > All the detailed data reported by the NDT tool comes from the 
> > Server-to-Client test.  This is because the web100 kernel captures 
> > the most details while sending.  Since the server has the web100 
> > enhanced kernel, the most details come when the server is sending 
> > data to the client.
> >
> > The message below indicates that the client is generating a large 
> > percentage of duplicate ACKs.  Duplicate ACKs occur when the client 
> > receives a packet out of order.  If the packet was really lost, the 
> > server would also record a large number of fast retransmits.  If 
> > the packets were just reordered, then the number of fast 
> > retransmits should be low.
> >
> > To diagnose this problem, it would really help to set the 
> > expectation level correctly.  Based on the fact that the S2C test 
> > is over 100 Mbps, can I safely assume that this is a gigabit 
> > Ethernet network?  The RTT value says that this is a LAN 
> > envrionment, but there is not a large queue beiing built up on the 
> > server.  The client is using a 64 KB buffer and window scaling is 
> > probably turned off.  Based on the RTT and TCP buffer size this 
> > connection should max out around 135 Mbps, which is what you are getting.
> >
> > One think you can try is to start the web100srv process with the 
> > "-r" option. This will cause it to log the web100 variable for the 
> > C2S test.  There might be something in there that can help you gain 
> > a little more understanding. Another thing to try is add the "-t" 
> > option to the web100srv process.  This will cause the server to 
> > generate tcpdump files for both s2c and c2s tests. These files can 
> > be analyzed to determine what is going on at the packet level.
> >
> > Finally, what kind of infrastructure is between the client and 
> > server?  How many switches and/or routers?  Have you tired moving 
> > the client to another switch port?  How about trying another 
> > Ethernet drop cable?
> >
> > Rich
> > At 05:33 PM 12/13/2007, Clarke Morledge wrote:
> > > I am trying to interpret some of the output from NDT (v5.4.8) that 
> > > indicates assymetrical performance variance.   The key component 
> > > for me is the line below:
> > > "No packet loss - but packets arrived out-of-order 12.58% of the time"
> > > But the problem I have is that it isn't clear to me where the 
> > > packets are arriving out-of-order.  In this case I am assuming 
> > > that it is from the client-to-server direction, but could there 
> > > also be some out-of-order in the server-to-client direction, 
> > > too?  The output isn't clear.
> > > However, is the packets arriving out-of-order component really 
> > > helping me to diagnose the assymetry problem, or is this a red 
> > > herring?  Is there some relevant web-100 variable(s) that I should 
> > > be looking at instead?
> > > Thanks.
> > > -----------------------
> > > WEB100 Enabled Statistics:
> > > Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
> > > checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
> > > running 10s outbound test (client-to-server [C2S]) . . . . . 40.08Mb/s
> > > running 10s inbound test (server-to-client [S2C]) . . . . . . 134.32Mb/s
> > >    ------  Client System Details  ------
> > > OS data: Name = Windows XP, Architecture = x86, Version = 5.1
> > > Java data: Vendor = Sun Microsystems Inc., Version = 1.6.0_02
> > >    ------  Web100 Detailed Analysis  ------
> > > 622 Mbps OC-12 link found.
> > > Link set to Full Duplex mode
> > > No network congestion discovered.
> > > Good network cable(s) found
> > > Normal duplex operation found.
> > > Web100 reports the Round trip time = 3.27 msec; the Packet size = 
> > > 1460 Bytes; and
> > > No packet loss - but packets arrived out-of-order 12.58% of the time
> > > C2S throughput test: Packet queuing detected: 0.00%
> > > S2C throughput test: Excessive packet queuing detected: 10.87%
> > > This connection is receiver limited 58.26% of the time.
> > >  Increasing the the client's receive buffer (63.0 KB) will 
> > > improve performance
> > > This connection is sender limited 41.27% of the time.
> >
> > ------------------------------------
> >
> >
> >
> > Richard A. Carlson                              e-mail: 
> >
> > Network Engineer                                phone:  (734) 352-7043
> > Internet2                                       fax:    (734) 913-4255
> > 1000 Oakbrook Dr; Suite 300
> > Ann Arbor,  MI  48104