ndt-users list created

[Rich Carlson <>]

Hi Nathalie;

I'm not sure I understand the configuration and the problem so let me 
ask for clarification.

You have 2 hosts connected back-to-back with a cross-over cable (fiber 
or copper?)  You have installed an NDT server on both nodes and from 
either node you get asymmetric results as shown below.  If this is not 
correct, then please clarify.

A couple of questions.
1) What Linux kernel version are you using?
2) what Myircom driver version are you using?
3) have you tuned any of the Myircom parameters?

more comments in-line

On 4/15/2010 1:27 PM, Gholmieh, Nathalie wrote:
> Hi-
>
> I have setup 2 NDT servers interconnected with a 10G link, both using
> Myricom 10G NICs, on our local network. the two servers have the same
> versions of NDT 3.6.1.
>
> when running NDT tests between the 2 servers, I get a C2S bandwidth of
> approximately 10Gbps, but the S2C bandwidth is not exceeding 3 Gbps, and
> that is on BOTH machines:
>
> [root@M
>  ~]# web100clt -n T -4 -l
>
> Testing network path for configuration and performance problems -- Using
> IPv4 address
>
> Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
>
> checking for firewalls . . . . . . . . . . . . . . . . . . . Done
>
> *running 10s outbound test (client to server) . . . . . 9351.25 Mb/s*
>
> *running 10s inbound test (server to client) . . . . . . 2605.19 Mb/s*

If the results in 1 direction showed these rates and a test in the 
opposite direction showed an inverted state (c2s lower than s2c) then I 
would suspect a problem with pacing or flow control in 1 direction or a 
configuration problem on 1 node.  However, if both nodes report the same 
results (c2s is always greater than s2c) then (1) it is a problem with 
my code in the xmit loop; or (2) an unknown problem.

> The slowest link in the end-to-end path is a 10 Gbps 10 Gigabit
> Ethernet/OC-192 subnet
>
> *Information [S2C]: Packet queuing detected: 72.52% (local buffers)*
>
> Server ‘T’ is not behind a firewall. [Connection to the ephemeral port
> was successful]
>
> Client is not behind a firewall. [Connection to the ephemeral port was
> successful]
>
> ------ Web100 Detailed Analysis ------
>
> Web100 reports the Round trip time = 4.09 msec;the Packet size = 8960
> Bytes; and

The RTT includes host queuing time (~5.6 MB in queue) using jumbo 
frames.  What is the txqueuelen value for this interface (ifconfig command)?

> There were 337 packets retransmitted, 9749 duplicate acks received, and
> 10089 SACK blocks received
>
> Packets arrived out-of-order 4.32% of the time.

Packets are being reordered.  This is probably due to the pkt processing 
by multiple cores.

> The connection stalled 1 times due to packet loss.
>
> The connection was idle 0.20 seconds (2.00%) of the time.

The sending node went through at least 1 timeout.  Add a 2nd -l to the 
command line and look at the last 3 variables that get reported (*CWND*) 
this will tell you the number of times TCP invoked the CA algorithm and 
what the high and low watermarks were.

> This connection is receiver limited 2.41% of the time.
>
> This connection is sender limited 76.06% of the time.

This is saying that the sender has limited resources, probably 
txqueuelen limits that prevent it from sending more data.  Note with 
jumbo frames 4 msec is about 5.6 MB and 624 packets.

> This connection is network limited 21.54% of the time.
>
> Web100 reports TCP negotiated the optional Performance Settings to:
>
> RFC 2018 Selective Acknowledgment: ON
>
> RFC 896 Nagle Algorithm: ON
>
> RFC 3168 Explicit Congestion Notification: OFF
>
> RFC 1323 Time Stamping: OFF
>
> RFC 1323 Window Scaling: ON; Scaling Factors - Server=9, Client=9
>
> The theoretical network limit is 2148.73 Mbps

This is from the Mathis equation ((pkt-size)/(rtt*sqrt(loss))).  This is 
about the same as the measured rate so this is the limiting factor.

> The NDT server has a 16384 KByte buffer which limits the throughput to
> 31295.84 Mbps
>
> Your PC/Workstation has a 12282 KByte buffer which limits the throughput
> to 23459.46 Mbps
>
> The network based flow control limits the throughput to 23533.01 Mbps

Buffer space from, your tuning parms below, is adequate.

> Client Data reports link is ' 9', Client Acks report link is ' 9'
>
> Server Data reports link is ' 9', Server Acks report link is ' 9'
>
> Packet size is preserved End-to-End
>
> Information: Network Address Translation (NAT) box is modifying the
> Server's IP address
>
> Server says [<IP>] but Client says [ T]
>
> Information: Network Address Translation (NAT) box is modifying the
> Client's IP address
>
> Server says [<IP2>] but Client says [M]
>
> [root@M
>  ~]#
>
> I have these sysctl values set on both servers:
>
> net.core.rmem_max = 16777216
>
> net.core.wmem_max = 16777216
>
> net.ipv4.tcp_wmem = 4096 65536 16777216
>
> net.ipv4.tcp_rmem = 4096 87380 16777216
>
> net.core.netdev_max_backlog = 250000
>
> net.ipv4.tcp_no_metrics_save = 1

Run the ifconfig command and report the txqueuelen value.

> I have also noticed that same asymmetry in the bandwidth while
> transferring an FTP file back and forth on the same link between the two
> servers.
>
> Note that the traffic both ways is using the same path.
>
> I am wondering why there is a difference between the sent and received
> bandwidth, and what parameters I should tune to use the full 10G both ways.
>
> Any ideas are very appreciated.
>
> Thanks!

I don't have a good clue right now.  Check the things listed 
(txqueuelen, version info, NIC tuning) and also run with more logging 
(-ll instead of -l).  Turning on flow control may help.  Also consider 
running an NPAD test.  The NPAD system probes for pkt queues and other 
system configuration settings and it may point out more details that can 
help you understand what is going on here.

Rich

> Nathalie~//