ndt-users list created

[Nat Stoddard <>]

Hi Rich,
Thank you for taking notice of this.
1.  I notice this behavior with both the web and the command line clients.  I 
only have the command line client to refer to when I am testing between my two 
NDT servers.  I get numbers very close to the same in both directions:
server1 to server2 outbound 8517.57 Mb/s, inbound 2164.96 Mb/s
server2 to server1 outbound 9101.33 Mb/s, inbound 2245.74 Mb/s

2. Top reports the CPU load on the server as high as 49%.  The client goes up to 
45%.

3.  I have pasted the web100clt -ll output below:

$ web100clt -n lblnet-test.lbl.gov -ll
Testing network path for configuration and performance problems  --  Using IPv4 
address
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
running 10s outbound test (client to server) . . . . .  9101.33 Mb/s
running 10s inbound test (server to client) . . . . . . 2245.74 Mb/s
The slowest link in the end-to-end path is a 2.4 Gbps OC-48 subnet
Information [S2C]: Packet queuing detected: 75.37% (remote buffers)
Server 'lblnet-test.lbl.gov' is not behind a firewall. [Connection to the 
ephemeral port was successful]
Client is not behind a firewall. [Connection to the ephemeral port was 
successful]

        ------  Web100 Detailed Analysis  ------

Web100 reports the Round trip time = 10.54 msec;the Packet size = 1448 Bytes; 
and
No packet loss was observed.
This connection is receiver limited 1.67% of the time.
This connection is sender limited 98.30% of the time.

    Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: ON
RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10
The theoretical network limit is 104855.13 Mbps
The NDT server has a 16384 KByte buffer which limits the throughput to 
12148.82 Mbps
Your PC/Workstation has a 12269 KByte buffer which limits the throughput to 
9097.53 Mbps
The network based flow control limits the throughput to 13645.63 Mbps

Client Data reports link is '  8', Client Acks report link is '  9'
Server Data reports link is '  9', Server Acks report link is '  9'
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Client IP addresses are preserved End-to-End
CurMSS: 1448
X_Rcvbuf: 87380
X_Sndbuf: 16777216
AckPktsIn: 267378
AckPktsOut: 0
BytesRetrans: 0
CongAvoid: 0
CongestionOverCount: 0
CongestionSignals: 0
CountRTT: 267379
CurCwnd: 18844272
CurRTO: 210
CurRwinRcvd: 12521472
CurRwinSent: 6144
CurSsthresh: 2147483647
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 2147483647
DataPktsIn: 0
DataPktsOut: 7873701
DupAcksIn: 0
ECNEnabled: 0
FastRetran: 0
MaxCwnd: 18844272
MaxMSS: 1448
MaxRTO: 212
MaxRTT: 12
MaxRwinRcvd: 12563456
MaxRwinSent: 6144
MaxSsthresh: 0
MinMSS: 1448
MinRTO: 201
MinRTT: 0
MinRwinRcvd: 6144
MinRwinSent: 5792
NagleEnabled: 1
OtherReductions: 0
PktsIn: 267378
PktsOut: 7873701
PktsRetrans: 0
RcvWinScale: 10
SACKEnabled: 3
SACKsRcvd: 0
SendStall: 0
SlowStart: 13012
SampleRTT: 10
SmoothedRTT: 10
SndWinScale: 10
SndLimTimeRwin: 167989
SndLimTimeCwnd: 2531
SndLimTimeSender: 9878037
SndLimTransRwin: 4723
SndLimTransCwnd: 42
SndLimTransSender: 4766
SndLimBytesRwin: 352300576
SndLimBytesCwnd: 1710336
SndLimBytesSender: 2147483647
SubsequentTimeouts: 0
SumRTT: 2817061
Timeouts: 0
TimestampsEnabled: 1
WinScaleRcvd: 10
WinScaleSent: 10
DupAcksOut: 0
StartTimeUsec: 6541
Duration: 10050267
c2sData: 8
c2sAck: 9
s2cData: 9
s2cAck: 9
half_duplex: 0
link: 100
congestion: 0
bad_cable: 0
mismatch: 0
spd: 1709.69
bw: 104855.13
loss: 0.000000000
avgrtt: 10.54
waitsec: 0.00
timesec: 10.00
order: 0.0000
rwintime: 0.0167
sendtime: 0.9830
cwndtime: 0.0003
rwin: 95.8516
swin: 128.0000
cwin: 143.7704
rttsec: 0.010536
Sndbuf: 16777216
aspd: 0.00000
CWND-Limited: 96729.00
minCWNDpeak: -1
maxCWNDpeak: -1
CWNDpeaks: -1


On 5/15/13 7:33 PM, Richard Carlson wrote:
> Byron;
>
> As you and Nat have noted, the NDT server seems to have a problem with 10 Gbps
> links.  I have noticed this behavior before, but I do not have any 10 Gbps 
> nodes
> to play with.
>
> If your willing to do some investigating, maybe we can figure out what's 
> going on.
>
> The questions I would start with are:
>
> 1) are you using the web client or the command line client?  (I think it's the
> command line client.)
>
> 2) what is the CPU load on the server and client while the tests are running.
> You can either run top with a short refresh time or turn the CPU monitoring 
> flag
> on.  (I don't expect the CPU is overloaded, but let's find out).
>
> 3) the NDT server captures the analysis data during the Server-to-Client test 
> so
> what does the more details page say?  (run the command line tool with the -ll
> option to extract this data or look at the web100srv.log file)
>
> However, looking at the code and the packet queuing message I think we have a
> probably answer.
>
> The packet queuing detected message is generated on the client during the
> analysis phase.  Both the client and the server measure the transfer speed
> (bytes transferred/time).  The client reports the value it calculated.  It is
> sent the speed the server calculated after the tests complete and the server
> sends this value as part of the larger test results back to the client for
> printing.  The formula is ((s2cspd-spdin)/s2cspd)*100 which is the servers
> calculation minus the clients calculation and converted to a percentage.  So
> your 80.16% says the server's calculation was 80% higher (a rough guess says 
> the
> server calculated the speed around 7.4 Gbps).
>
> Given all this I'd look at the command line client's read data loop. It's not
> terminating properly and in this case ran about 50 seconds instead of 10.
>
> Try running the command line tool with some -d flags to print out debugging
> information.  That might tell you more about what's going on.
>
> Rich
>
> On 05/15/2013 04:25 PM, Byron Hicks wrote:
> > I'm reasonably certain that I have a clean path.
> >
> > Both boxes are running NDT, and I get the same result in both directions:
> >
> > Houston:
> >
> > running 10s outbound test (client to server) . . . . .  9123.91 Mb/s
> > running 10s inbound test (server to client) . . . . . . 1434.11 Mb/s
> >
> > Dallas:
> >
> > running 10s outbound test (client to server) . . . . .  8953.05 Mb/s
> > running 10s inbound test (server to client) . . . . . . 1440.18 Mb/s
> >
> > If it were a traffic loss issue, I would expect that the
> > outbound/inbound numbers would flip, with the lower number being on the
> > "leg" of the duplex path that had the traffic loss.
> >
> > But I'm not.  Client to Server is 9Gb/s and Server to Client is 1.4Gb/s,
> > regardless of which NDT server I'm testing from/to.  And considering
> > that I'm getting 9Gb/s on a 10Gb/s link using iperf in both directions,
> > I'm pretty sure packet loss is a not a factor.
> >
> > How do I interpret the following:
> >
> > Information [S2C]: Packet queuing detected: 80.16% (remote buffers)
> >
> > Where is the packet queuing happening?
> >
> >
> > On 05/15/2013 01:37 PM, Brian Tierney wrote:
> > > Another possibility is that I've seen cases where, on a path with packet
> > > loss, different clients seem to trigger different loss patterns.
> > >
> > > For example, here is on a clean path:
> > >
> > >
> > > > web100clt -n ps-lax-10g.cenic.net -b 33554432
> > > running 10s outbound test (client to server) . . . . .  2321.11 Mb/s
> > > running 10s inbound test (server to client) . . . . . . 2802.95 Mb/s
> > >
> > > vs bwctl:
> > >
> > > > bwctl -c ps-lax-10g.cenic.net -fm
> > > bwctl: Using tool: iperf
> > > [ 14] local 137.164.28.105 port 5001 connected with 198.129.254.98 port 5001
> > > [ ID] Interval       Transfer     Bandwidth
> > > [ 14]  0.0-10.0 sec  2984 MBytes  2496 Mbits/sec
> > >
> > > performance is similar.
> > >
> > > ----------
> > >
> > > And here are the results for a path with packet loss:
> > >
> > > > web100clt -n ps-lax-10g.cenic.net -b 33554432
> > > running 10s outbound test (client to server) . . . . .  18.06 Mb/s
> > > running 10s inbound test (server to client) . . . . . . 2492.69 Mb/s
> > >
> > > > bwctl -c ps-lax-10g.cenic.net -fm
> > > [ 14] local 137.164.28.105 port 5001 connected with 198.129.254.150 port 5001
> > > [ ID] Interval       Transfer     Bandwidth
> > > [ 14]  0.0-10.3 sec   552 MBytes   450 Mbits/sec
> > >
> > > Here iperf does 30x better than NDT (and btw, nuttcp results agree with the
> > > NDT results in this case)
> > >
> > > My guess is that different tools have different burst characteristics, and
> > > these trigger different amounts of packet loss.

--
Nathaniel T. Stoddard
LBLnet Services Group
Lawrence Berkeley Lab