ndt-users list created

["Gholmieh, Nathalie" <>]

Hi-

 

I have setup 2 NDT servers interconnected with a 10G link, both using Myricom 10G NICs, on our local network.  the two servers have the same versions of NDT 3.6.1.

 

when running NDT tests between the 2 servers, I get a C2S bandwidth of approximately 10Gbps, but the S2C bandwidth is not exceeding 3 Gbps, and that is on BOTH machines:

 

[root@M ~]# web100clt -n T -4 -l

Testing network path for configuration and performance problems  --  Using IPv4 address

Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done

checking for firewalls . . . . . . . . . . . . . . . . . . .  Done

running 10s outbound test (client to server) . . . . .  9351.25 Mb/s

running 10s inbound test (server to client) . . . . . . 2605.19 Mb/s

The slowest link in the end-to-end path is a 10 Gbps 10 Gigabit Ethernet/OC-192 subnet

Information [S2C]: Packet queuing detected: 72.52% (local buffers)

Server ‘T’ is not behind a firewall. [Connection to the ephemeral port was successful]

Client is not behind a firewall. [Connection to the ephemeral port was successful]

 

        ------  Web100 Detailed Analysis  ------

 

Web100 reports the Round trip time = 4.09 msec;the Packet size = 8960 Bytes; and

There were 337 packets retransmitted, 9749 duplicate acks received, and 10089 SACK blocks received

Packets arrived out-of-order 4.32% of the time.

The connection stalled 1 times due to packet loss.

The connection was idle 0.20 seconds (2.00%) of the time.

This connection is receiver limited 2.41% of the time.

This connection is sender limited 76.06% of the time.

This connection is network limited 21.54% of the time.

 

    Web100 reports TCP negotiated the optional Performance Settings to:

RFC 2018 Selective Acknowledgment: ON

RFC 896 Nagle Algorithm: ON

RFC 3168 Explicit Congestion Notification: OFF

RFC 1323 Time Stamping: OFF

RFC 1323 Window Scaling: ON; Scaling Factors - Server=9, Client=9

The theoretical network limit is 2148.73 Mbps

The NDT server has a 16384 KByte buffer which limits the throughput to 31295.84 Mbps

Your PC/Workstation has a 12282 KByte buffer which limits the throughput to 23459.46 Mbps

The network based flow control limits the throughput to 23533.01 Mbps

 

Client Data reports link is '  9', Client Acks report link is '  9'

Server Data reports link is '  9', Server Acks report link is '  9'

Packet size is preserved End-to-End

Information: Network Address Translation (NAT) box is modifying the Server's IP address

        Server says [<IP>] but Client says [ T]

Information: Network Address Translation (NAT) box is modifying the Client's IP address

        Server says [<IP2>] but Client says [M]

[root@M ~]#

 

I have these sysctl values set on both servers:

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216

net.core.netdev_max_backlog = 250000

net.ipv4.tcp_no_metrics_save = 1

 

I have also noticed that same asymmetry in the bandwidth while transferring an FTP file back and forth on the same link between the two servers.

 

Note that the traffic both ways is using the same path.

 

I am wondering why there is a difference between the sent and received bandwidth, and what parameters I should tune to use the full 10G both ways.

 

Any ideas are very appreciated.

 

Thanks!

Nathalie~