Skip to Content.
Sympa Menu

ndt-users - Re: 10GigE ndt server (3.6.3) issues

Subject: ndt-users list created

List archive

Re: 10GigE ndt server (3.6.3) issues


Chronological Thread 
  • From: Rich Carlson <>
  • To: "Hao, Justin C" <>
  • Cc: "" <>
  • Subject: Re: 10GigE ndt server (3.6.3) issues
  • Date: Thu, 30 Sep 2010 10:23:31 -0400

Hi Justin;

Please run the tests with the -ll option and post the results to this list or just send them to me. Off the top of my head I don't know why the results would be different based on the direction.

Rich

On 9/29/2010 2:59 PM, Hao, Justin C wrote:
Howdy Rich,

I've got the boxes connected to each other, Server 1 is running ndt 3.6.3 on
CentOS 5.5 (2.6.35-web100 kernel) and Server 2 is running Perfsonar 3.2rc1
(looks like ndt 3.4.4a)

I'm still seeing the asynch C2S/S2C values and was wondering if you could
shed any light/point me in the right direction. I've included snapshots of
the web100clt results for each server, please let me know if you need
additional information. I've configured both servers identically in terms of
sysctl.conf tcp settings as well as txqueuelen of 10000 and MTU of 9000

Server 1(10.0.0.1) to Server 2(10.0.0.2)
[root@dhcp-135-164
etc]# web100clt -4 -n 10.0.0.2
Testing network path for configuration and performance problems -- Using
IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
checking for firewalls . . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client to server) . . . . . 9876.02 Mb/s
running 10s inbound test (server to client) . . . . . . 3041.14 Mb/s
Server unable to determine bottleneck link type.
Information [S2C]: Packet queuing detected: 69.33% (local buffers)
Server '10.0.0.2' is not behind a firewall. [Connection to the ephemeral port
was successful]
Client is not behind a firewall. [Connection to the ephemeral port was
successful]
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Client IP addresses are preserved End-to-End
[root@dhcp-135-164
etc]#

Server 2(10.0.0.2) to Server 1(10.0.0.1)
[root@localhost
etc]# web100clt -4 -n 10.0.0.1
Testing network path for configuration and performance problems -- Using
IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
checking for firewalls . . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client to server) . . . . . 9898.48 Mb/s
running 10s inbound test (server to client) . . . . . . 2622.73 Mb/s
Server unable to determine bottleneck link type.
Information: Other network traffic is congesting the link
Information [S2C]: Packet queuing detected: 72.39% (remote buffers)
Server '10.0.0.1' is not behind a firewall. [Connection to the ephemeral port
was successful]
Client is not behind a firewall. [Connection to the ephemeral port was
successful]
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Client IP addresses are preserved End-to-End
[root@localhost
etc]#


-----
Justin Hao
CCNA
Network Engineer, ITS Networking
The University of Texas at Austin

-----

On Sep 29, 2010, at 9:01 AM, Hao, Justin C wrote:

that was step two, i'm going to connect the two servers to each other and see
what i can see

-----
Justin Hao
CCNA
Network Engineer, ITS Networking
The University of Texas at Austin

-----

On Sep 29, 2010, at 9:00 AM, Rich Carlson wrote:

Justin;

Use the latest JRE, but look at the console service. On Win based
machines I've found the upgrade process doesn't remove old version of
the java console. You need to manually do this through the control
panel. I don't have a linux box handy to see what it does.

In any case, I'd say you aren't really testing the NDT tool on a
loopback interface, so running the server and client on a single machine
isn't really going to tell you much. Bring up a server and use the
applet to test a number of different clients to see what is going on.

Rich

On 9/29/2010 9:37 AM, Hao, Justin C wrote:
Howdy Rich,

I've tried with both the command line and the web client, only the web client
has returned the negative values, but the command line client has also
returned a wide range of results. what is the recommended java RE to use?

-----
Justin Hao
CCNA
Network Engineer, ITS Networking
The University of Texas at Austin

-----

On Sep 29, 2010, at 8:30 AM, Rich Carlson wrote:

Justin;

The Web100 system uses 32bit counters, so I suspect the negative speeds
come from the use of signed int vars instead of unsigned.

When testing on the local box, the traffic goes through the loopback
interface (lo0) instead of the NIC. This means you are testing the OS
and it's memory management system more that anything else.
Changing/setting variables on the ethx interface will have no effect.

You should not set the tcp_mem value to 16 M. The value for this
variable is in pages, NOT bytes. See
Documentation/networking/ip-sysctl.txt doc in the kernel source tree for
more details.

tcp_mem - vector of 3 INTEGERs: min, pressure, max
min: below this number of pages TCP is not bothered about its
memory appetite.

pressure: when amount of memory allocated by TCP exceeds this
number
of pages, TCP moderates its memory consumption and enters memory
pressure mode, which is exited when memory consumption falls
under "min".

max: number of pages allowed for queueing by all TCP sockets.

Defaults are calculated at boot time from amount of available
memory.


Are you using the command line client (web100clt) or the Java Applet
(via the browser) to run these tests? It was recently reported that a
site was getting high variability in the S2C tests with the Java client.
It turned out that they had 2 Java consoles installed. Removing the
older console cleared up the problem.

Regards;
Rich

On 9/27/2010 6:49 PM, Hao, Justin C wrote:
So i'm setting up a pair of ndt test servers for our new datacenter and
running into some hurdles.

First off, i haven't connected them to each other (or to a 10Gig network), so
all my testing has been on a single box running loopback tests to itself (i
have no other 10GigE hosts available to me at the moment).

I'm running CentOS 5.5 and patched kernel 2.6.35 with the proper web100
version etc.

It's currently hooked up via 1GigE and i've run several different clients
against it and I get good results for 1Gig performance

I'm running loopback tests to tweak TCP settings while I wait for our 10Gig
environment to be made ready. I'm getting 15-18Gig/s for C2S but for S2C i'm
getting numbers all over the place. from 3.5Gig/s to 500Mb/s. and most
particularly odd, i'm seeing negative numbers in some of the test output.

I welcome any comments and suggestions for tuning this server (it's a dell
r610 w/ an intel 10GigE adapter)

note: I've also configured the ethernet interface to use 9000 MTU and a
txqueuelen of 10000

Server TCP settings:

# increase TCP max buffer size setable using setsockopt()

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to 16MB for 1GE, and 32M or 54M for 10GE

net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.tcp_rmem = 10240 87380 16777216
net.ipv4.tcp_wmem = 10240 65536 16777216

net.ipv4.tcp_window_scaling = 1

# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1

# recommended to increase this for 10G NICS
net.core.netdev_max_backlog = 262144

NDT Output:

TCP/Web100 Network Diagnostic Tool v3.6.3
Click START to start the test

** Starting test 1 of 1 **
Connecting to '127.0.0.1' [/127.0.0.1] to run test
Connected to: 127.0.0.1-- Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done.
Checking for firewalls . . . . . . . . . . . . . . . . . . . Done.
running 10s outbound test (client-to-server [C2S]) . . . . . 13744.22Mb/s
running 10s inbound test (server-to-client [S2C]) . . . . . . -83702.57kb/s
Server unable to determine bottleneck link type.
[S2C]: Packet queueing detected

Click START to re-test

** Starting test 1 of 1 **
Connecting to '127.0.0.1' [/127.0.0.1] to run test
Connected to: 127.0.0.1-- Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done.
Checking for firewalls . . . . . . . . . . . . . . . . . . . Done.
running 10s outbound test (client-to-server [C2S]) . . . . . 12876.05Mb/s
running 10s inbound test (server-to-client [S2C]) . . . . . . 1006.86Mb/s
Server unable to determine bottleneck link type.
[S2C]: Packet queueing detected

Click START to re-test

** Starting test 1 of 1 **
Connecting to '127.0.0.1' [/127.0.0.1] to run test
Connected to: 127.0.0.1-- Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done.
Checking for firewalls . . . . . . . . . . . . . . . . . . . Done.
running 10s outbound test (client-to-server [C2S]) . . . . . 18466.0Mb/s
running 10s inbound test (server-to-client [S2C]) . . . . . . -1710004.63kb/s
Server unable to determine bottleneck link type.
[S2C]: Packet queueing detected

Click START to re-test


-----
Justin Hao
CCNA
Network Engineer, ITS Networking
The University of Texas at Austin

-----










Archive powered by MHonArc 2.6.16.

Top of Page