ndt-users list created

[Richard Carlson <>]

Nat;

This is good.  As I thought, the CPU isn't the bottleneck.

The connection is sender limited, meaning its' not network or receiver 
limited.  The send/receive buffers don't limit the throughput.

The pkt-pair timing say OC-48 to 10 GE so that's good.

So lets look at the raw numbers.
        CurMSS = 1448 (normal Ethernet frame)
        Data Pkts Out = 7873701
        Data Bytes Out = 2,147,483,647
        Timesec = 10.00 sec

This is suspicious.
Bytes sent should be: 7,873,701 x 1448 = 11,401,119,048
Speed calculates to 9.1 Gbps  So this is what the server acheived, but 
not what is logged by the client.

But the server report 2.2 GB which converts to 1.7 Gbps. which is what 
the spd: variable says.

I wonder if this is just a variable overflow error.  The DataBytesOut 
variable is a 32 bit counter.  There is an HCDataOctetsOut variable (a 
64 bit counter)

So, this is a problem that may impact some of the analysis messages. 
line 1082 in web100srv.c sets the variable spd based on the DataBytesOut 
value.  Either the code should grab the HCDataOctetsOut value or a test 
should be made and if s2cspd is greater that something like 5 Gbps, use 
the HC value otherwise use the existing value.

I'm looking at the code and it show the server is able to right log 
messages that might help here.  Restart the server with 1 or 2 debug 
flags (-d or -dd) and post the output here.

Got it !!!!!!  While the DataByteOut vs HCDataOctetOut issue noted above 
is a potential bug, it is not causing this bug.

The file test_s2c_clt.c is the culprit.  The data read subroutine 
declares the byte counter as an uint32_t.  This is too small for the 
amount of data we are getting at 10 Gbps, and it is overflowing. 
Changing that to a uint64_t variable should fix this problem.  The 
server uses double, so it doesn't overflow.

So change line 32 of test_s2c_clt.c  s/uint32_t/uint64_t
and recompile.  See if that fixes this bug.

Same with the Java client, line 1253 of the Tcpbw100.java file declares 
bytes as an int, this should be a double to prevent counter overflows at 
10 Gbps.

Rich

On 05/16/2013 01:30 PM, Nat Stoddard wrote:
> Hi Rich,
> Thank you for taking notice of this.
> 1.  I notice this behavior with both the web and the command line
> clients.  I only have the command line client to refer to when I am
> testing between my two NDT servers.  I get numbers very close to the
> same in both directions:
> server1 to server2 outbound 8517.57 Mb/s, inbound 2164.96 Mb/s
> server2 to server1 outbound 9101.33 Mb/s, inbound 2245.74 Mb/s
>
> 2. Top reports the CPU load on the server as high as 49%.  The client
> goes up to 45%.
>
> 3.  I have pasted the web100clt -ll output below:
>
> $ web100clt -n lblnet-test.lbl.gov -ll
> Testing network path for configuration and performance problems  --
> Using IPv4 address
> Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
> checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
> running 10s outbound test (client to server) . . . . .  9101.33 Mb/s
> running 10s inbound test (server to client) . . . . . . 2245.74 Mb/s
> The slowest link in the end-to-end path is a 2.4 Gbps OC-48 subnet
> Information [S2C]: Packet queuing detected: 75.37% (remote buffers)
> Server 'lblnet-test.lbl.gov' is not behind a firewall. [Connection to
> the ephemeral port was successful]
> Client is not behind a firewall. [Connection to the ephemeral port was
> successful]
>
>      ------  Web100 Detailed Analysis  ------
>
> Web100 reports the Round trip time = 10.54 msec;the Packet size = 1448
> Bytes; and
> No packet loss was observed.
> This connection is receiver limited 1.67% of the time.
> This connection is sender limited 98.30% of the time.
>
>      Web100 reports TCP negotiated the optional Performance Settings to:
> RFC 2018 Selective Acknowledgment: ON
> RFC 896 Nagle Algorithm: ON
> RFC 3168 Explicit Congestion Notification: OFF
> RFC 1323 Time Stamping: ON
> RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10
> The theoretical network limit is 104855.13 Mbps
> The NDT server has a 16384 KByte buffer which limits the throughput to
> 12148.82 Mbps
> Your PC/Workstation has a 12269 KByte buffer which limits the throughput
> to 9097.53 Mbps
> The network based flow control limits the throughput to 13645.63 Mbps
>
> Client Data reports link is '  8', Client Acks report link is '  9'
> Server Data reports link is '  9', Server Acks report link is '  9'
> Packet size is preserved End-to-End
> Server IP addresses are preserved End-to-End
> Client IP addresses are preserved End-to-End
> CurMSS: 1448
> X_Rcvbuf: 87380
> X_Sndbuf: 16777216
> AckPktsIn: 267378
> AckPktsOut: 0
> BytesRetrans: 0
> CongAvoid: 0
> CongestionOverCount: 0
> CongestionSignals: 0
> CountRTT: 267379
> CurCwnd: 18844272
> CurRTO: 210
> CurRwinRcvd: 12521472
> CurRwinSent: 6144
> CurSsthresh: 2147483647
> DSACKDups: 0
> DataBytesIn: 0
> DataBytesOut: 2147483647
> DataPktsIn: 0
> DataPktsOut: 7873701
> DupAcksIn: 0
> ECNEnabled: 0
> FastRetran: 0
> MaxCwnd: 18844272
> MaxMSS: 1448
> MaxRTO: 212
> MaxRTT: 12
> MaxRwinRcvd: 12563456
> MaxRwinSent: 6144
> MaxSsthresh: 0
> MinMSS: 1448
> MinRTO: 201
> MinRTT: 0
> MinRwinRcvd: 6144
> MinRwinSent: 5792
> NagleEnabled: 1
> OtherReductions: 0
> PktsIn: 267378
> PktsOut: 7873701
> PktsRetrans: 0
> RcvWinScale: 10
> SACKEnabled: 3
> SACKsRcvd: 0
> SendStall: 0
> SlowStart: 13012
> SampleRTT: 10
> SmoothedRTT: 10
> SndWinScale: 10
> SndLimTimeRwin: 167989
> SndLimTimeCwnd: 2531
> SndLimTimeSender: 9878037
> SndLimTransRwin: 4723
> SndLimTransCwnd: 42
> SndLimTransSender: 4766
> SndLimBytesRwin: 352300576
> SndLimBytesCwnd: 1710336
> SndLimBytesSender: 2147483647
> SubsequentTimeouts: 0
> SumRTT: 2817061
> Timeouts: 0
> TimestampsEnabled: 1
> WinScaleRcvd: 10
> WinScaleSent: 10
> DupAcksOut: 0
> StartTimeUsec: 6541
> Duration: 10050267
> c2sData: 8
> c2sAck: 9
> s2cData: 9
> s2cAck: 9
> half_duplex: 0
> link: 100
> congestion: 0
> bad_cable: 0
> mismatch: 0
> spd: 1709.69
> bw: 104855.13
> loss: 0.000000000
> avgrtt: 10.54
> waitsec: 0.00
> timesec: 10.00
> order: 0.0000
> rwintime: 0.0167
> sendtime: 0.9830
> cwndtime: 0.0003
> rwin: 95.8516
> swin: 128.0000
> cwin: 143.7704
> rttsec: 0.010536
> Sndbuf: 16777216
> aspd: 0.00000
> CWND-Limited: 96729.00
> minCWNDpeak: -1
> maxCWNDpeak: -1
> CWNDpeaks: -1
>
>
> On 5/15/13 7:33 PM, Richard Carlson wrote:
> > Byron;
> >
> > As you and Nat have noted, the NDT server seems to have a problem with
> > 10 Gbps
> > links.  I have noticed this behavior before, but I do not have any 10
> > Gbps nodes
> > to play with.
> >
> > If your willing to do some investigating, maybe we can figure out
> > what's going on.
> >
> > The questions I would start with are:
> >
> > 1) are you using the web client or the command line client?  (I think
> > it's the
> > command line client.)
> >
> > 2) what is the CPU load on the server and client while the tests are
> > running.
> > You can either run top with a short refresh time or turn the CPU
> > monitoring flag
> > on.  (I don't expect the CPU is overloaded, but let's find out).
> >
> > 3) the NDT server captures the analysis data during the
> > Server-to-Client test so
> > what does the more details page say?  (run the command line tool with
> > the -ll
> > option to extract this data or look at the web100srv.log file)
> >
> > However, looking at the code and the packet queuing message I think we
> > have a
> > probably answer.
> >
> > The packet queuing detected message is generated on the client during the
> > analysis phase.  Both the client and the server measure the transfer
> > speed
> > (bytes transferred/time).  The client reports the value it
> > calculated.  It is
> > sent the speed the server calculated after the tests complete and the
> > server
> > sends this value as part of the larger test results back to the client
> > for
> > printing.  The formula is ((s2cspd-spdin)/s2cspd)*100 which is the
> > servers
> > calculation minus the clients calculation and converted to a
> > percentage.  So
> > your 80.16% says the server's calculation was 80% higher (a rough
> > guess says the
> > server calculated the speed around 7.4 Gbps).
> >
> > Given all this I'd look at the command line client's read data loop.
> > It's not
> > terminating properly and in this case ran about 50 seconds instead of 10.
> >
> > Try running the command line tool with some -d flags to print out
> > debugging
> > information.  That might tell you more about what's going on.
> >
> > Rich
> >
> > On 05/15/2013 04:25 PM, Byron Hicks wrote:
> > > I'm reasonably certain that I have a clean path.
> > >
> > > Both boxes are running NDT, and I get the same result in both
> > > directions:
> > >
> > > Houston:
> > >
> > > running 10s outbound test (client to server) . . . . .  9123.91 Mb/s
> > > running 10s inbound test (server to client) . . . . . . 1434.11 Mb/s
> > >
> > > Dallas:
> > >
> > > running 10s outbound test (client to server) . . . . .  8953.05 Mb/s
> > > running 10s inbound test (server to client) . . . . . . 1440.18 Mb/s
> > >
> > > If it were a traffic loss issue, I would expect that the
> > > outbound/inbound numbers would flip, with the lower number being on the
> > > "leg" of the duplex path that had the traffic loss.
> > >
> > > But I'm not.  Client to Server is 9Gb/s and Server to Client is 1.4Gb/s,
> > > regardless of which NDT server I'm testing from/to.  And considering
> > > that I'm getting 9Gb/s on a 10Gb/s link using iperf in both directions,
> > > I'm pretty sure packet loss is a not a factor.
> > >
> > > How do I interpret the following:
> > >
> > > Information [S2C]: Packet queuing detected: 80.16% (remote buffers)
> > >
> > > Where is the packet queuing happening?
> > >
> > >
> > > On 05/15/2013 01:37 PM, Brian Tierney wrote:
> > > > Another possibility is that I've seen cases where, on a path with
> > > > packet
> > > > loss, different clients seem to trigger different loss patterns.
> > > >
> > > > For example, here is on a clean path:
> > > >
> > > >
> > > > > web100clt -n ps-lax-10g.cenic.net -b 33554432
> > > > running 10s outbound test (client to server) . . . . .  2321.11 Mb/s
> > > > running 10s inbound test (server to client) . . . . . . 2802.95 Mb/s
> > > >
> > > > vs bwctl:
> > > >
> > > > > bwctl -c ps-lax-10g.cenic.net -fm
> > > > bwctl: Using tool: iperf
> > > > [ 14] local 137.164.28.105 port 5001 connected with 198.129.254.98
> > > > port 5001
> > > > [ ID] Interval       Transfer     Bandwidth
> > > > [ 14]  0.0-10.0 sec  2984 MBytes  2496 Mbits/sec
> > > >
> > > > performance is similar.
> > > >
> > > > ----------
> > > >
> > > > And here are the results for a path with packet loss:
> > > >
> > > > > web100clt -n ps-lax-10g.cenic.net -b 33554432
> > > > running 10s outbound test (client to server) . . . . .  18.06 Mb/s
> > > > running 10s inbound test (server to client) . . . . . . 2492.69 Mb/s
> > > >
> > > > > bwctl -c ps-lax-10g.cenic.net -fm
> > > > [ 14] local 137.164.28.105 port 5001 connected with 198.129.254.150
> > > > port 5001
> > > > [ ID] Interval       Transfer     Bandwidth
> > > > [ 14]  0.0-10.3 sec   552 MBytes   450 Mbits/sec
> > > >
> > > > Here iperf does 30x better than NDT (and btw, nuttcp results agree
> > > > with the
> > > > NDT results in this case)
> > > >
> > > > My guess is that different tools have different burst
> > > > characteristics, and
> > > > these trigger different amounts of packet loss.