ndt-users list created

[Byron Hicks <>]

Rich,

Do you still need me to do more testing on my end, or was the
information provided by Nat Stoddard enough?


On 05/16/2013 10:39 PM, Richard Carlson wrote:
> Nat;
> 
> This is good.  As I thought, the CPU isn't the bottleneck.
> 
> The connection is sender limited, meaning its' not network or receiver
> limited.  The send/receive buffers don't limit the throughput.
> 
> The pkt-pair timing say OC-48 to 10 GE so that's good.
> 
> So lets look at the raw numbers.
>     CurMSS = 1448 (normal Ethernet frame)
>     Data Pkts Out = 7873701
>     Data Bytes Out = 2,147,483,647
>     Timesec = 10.00 sec
> 
> This is suspicious.
> Bytes sent should be: 7,873,701 x 1448 = 11,401,119,048
> Speed calculates to 9.1 Gbps  So this is what the server acheived, but
> not what is logged by the client.
> 
> But the server report 2.2 GB which converts to 1.7 Gbps. which is what
> the spd: variable says.
> 
> I wonder if this is just a variable overflow error.  The DataBytesOut
> variable is a 32 bit counter.  There is an HCDataOctetsOut variable (a
> 64 bit counter)
> 
> So, this is a problem that may impact some of the analysis messages.
> line 1082 in web100srv.c sets the variable spd based on the DataBytesOut
> value.  Either the code should grab the HCDataOctetsOut value or a test
> should be made and if s2cspd is greater that something like 5 Gbps, use
> the HC value otherwise use the existing value.
> 
> I'm looking at the code and it show the server is able to right log
> messages that might help here.  Restart the server with 1 or 2 debug
> flags (-d or -dd) and post the output here.
> 
> Got it !!!!!!  While the DataByteOut vs HCDataOctetOut issue noted above
> is a potential bug, it is not causing this bug.
> 
> The file test_s2c_clt.c is the culprit.  The data read subroutine
> declares the byte counter as an uint32_t.  This is too small for the
> amount of data we are getting at 10 Gbps, and it is overflowing.
> Changing that to a uint64_t variable should fix this problem.  The
> server uses double, so it doesn't overflow.
> 
> So change line 32 of test_s2c_clt.c  s/uint32_t/uint64_t
> and recompile.  See if that fixes this bug.
> 
> Same with the Java client, line 1253 of the Tcpbw100.java file declares
> bytes as an int, this should be a double to prevent counter overflows at
> 10 Gbps.
> 
> Rich
> 
> On 05/16/2013 01:30 PM, Nat Stoddard wrote:
>> Hi Rich,
>> Thank you for taking notice of this.
>> 1.  I notice this behavior with both the web and the command line
>> clients.  I only have the command line client to refer to when I am
>> testing between my two NDT servers.  I get numbers very close to the
>> same in both directions:
>> server1 to server2 outbound 8517.57 Mb/s, inbound 2164.96 Mb/s
>> server2 to server1 outbound 9101.33 Mb/s, inbound 2245.74 Mb/s
>>
>> 2. Top reports the CPU load on the server as high as 49%.  The client
>> goes up to 45%.
>>
>> 3.  I have pasted the web100clt -ll output below:
>>
>> $ web100clt -n lblnet-test.lbl.gov -ll
>> Testing network path for configuration and performance problems  --
>> Using IPv4 address
>> Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
>> checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
>> running 10s outbound test (client to server) . . . . .  9101.33 Mb/s
>> running 10s inbound test (server to client) . . . . . . 2245.74 Mb/s
>> The slowest link in the end-to-end path is a 2.4 Gbps OC-48 subnet
>> Information [S2C]: Packet queuing detected: 75.37% (remote buffers)
>> Server 'lblnet-test.lbl.gov' is not behind a firewall. [Connection to
>> the ephemeral port was successful]
>> Client is not behind a firewall. [Connection to the ephemeral port was
>> successful]
>>
>>      ------  Web100 Detailed Analysis  ------
>>
>> Web100 reports the Round trip time = 10.54 msec;the Packet size = 1448
>> Bytes; and
>> No packet loss was observed.
>> This connection is receiver limited 1.67% of the time.
>> This connection is sender limited 98.30% of the time.
>>
>>      Web100 reports TCP negotiated the optional Performance Settings to:
>> RFC 2018 Selective Acknowledgment: ON
>> RFC 896 Nagle Algorithm: ON
>> RFC 3168 Explicit Congestion Notification: OFF
>> RFC 1323 Time Stamping: ON
>> RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10
>> The theoretical network limit is 104855.13 Mbps
>> The NDT server has a 16384 KByte buffer which limits the throughput to
>> 12148.82 Mbps
>> Your PC/Workstation has a 12269 KByte buffer which limits the throughput
>> to 9097.53 Mbps
>> The network based flow control limits the throughput to 13645.63 Mbps
>>
>> Client Data reports link is '  8', Client Acks report link is '  9'
>> Server Data reports link is '  9', Server Acks report link is '  9'
>> Packet size is preserved End-to-End
>> Server IP addresses are preserved End-to-End
>> Client IP addresses are preserved End-to-End
>> CurMSS: 1448
>> X_Rcvbuf: 87380
>> X_Sndbuf: 16777216
>> AckPktsIn: 267378
>> AckPktsOut: 0
>> BytesRetrans: 0
>> CongAvoid: 0
>> CongestionOverCount: 0
>> CongestionSignals: 0
>> CountRTT: 267379
>> CurCwnd: 18844272
>> CurRTO: 210
>> CurRwinRcvd: 12521472
>> CurRwinSent: 6144
>> CurSsthresh: 2147483647
>> DSACKDups: 0
>> DataBytesIn: 0
>> DataBytesOut: 2147483647
>> DataPktsIn: 0
>> DataPktsOut: 7873701
>> DupAcksIn: 0
>> ECNEnabled: 0
>> FastRetran: 0
>> MaxCwnd: 18844272
>> MaxMSS: 1448
>> MaxRTO: 212
>> MaxRTT: 12
>> MaxRwinRcvd: 12563456
>> MaxRwinSent: 6144
>> MaxSsthresh: 0
>> MinMSS: 1448
>> MinRTO: 201
>> MinRTT: 0
>> MinRwinRcvd: 6144
>> MinRwinSent: 5792
>> NagleEnabled: 1
>> OtherReductions: 0
>> PktsIn: 267378
>> PktsOut: 7873701
>> PktsRetrans: 0
>> RcvWinScale: 10
>> SACKEnabled: 3
>> SACKsRcvd: 0
>> SendStall: 0
>> SlowStart: 13012
>> SampleRTT: 10
>> SmoothedRTT: 10
>> SndWinScale: 10
>> SndLimTimeRwin: 167989
>> SndLimTimeCwnd: 2531
>> SndLimTimeSender: 9878037
>> SndLimTransRwin: 4723
>> SndLimTransCwnd: 42
>> SndLimTransSender: 4766
>> SndLimBytesRwin: 352300576
>> SndLimBytesCwnd: 1710336
>> SndLimBytesSender: 2147483647
>> SubsequentTimeouts: 0
>> SumRTT: 2817061
>> Timeouts: 0
>> TimestampsEnabled: 1
>> WinScaleRcvd: 10
>> WinScaleSent: 10
>> DupAcksOut: 0
>> StartTimeUsec: 6541
>> Duration: 10050267
>> c2sData: 8
>> c2sAck: 9
>> s2cData: 9
>> s2cAck: 9
>> half_duplex: 0
>> link: 100
>> congestion: 0
>> bad_cable: 0
>> mismatch: 0
>> spd: 1709.69
>> bw: 104855.13
>> loss: 0.000000000
>> avgrtt: 10.54
>> waitsec: 0.00
>> timesec: 10.00
>> order: 0.0000
>> rwintime: 0.0167
>> sendtime: 0.9830
>> cwndtime: 0.0003
>> rwin: 95.8516
>> swin: 128.0000
>> cwin: 143.7704
>> rttsec: 0.010536
>> Sndbuf: 16777216
>> aspd: 0.00000
>> CWND-Limited: 96729.00
>> minCWNDpeak: -1
>> maxCWNDpeak: -1
>> CWNDpeaks: -1
>>
>>
>> On 5/15/13 7:33 PM, Richard Carlson wrote:
>>> Byron;
>>>
>>> As you and Nat have noted, the NDT server seems to have a problem with
>>> 10 Gbps
>>> links.  I have noticed this behavior before, but I do not have any 10
>>> Gbps nodes
>>> to play with.
>>>
>>> If your willing to do some investigating, maybe we can figure out
>>> what's going on.
>>>
>>> The questions I would start with are:
>>>
>>> 1) are you using the web client or the command line client?  (I think
>>> it's the
>>> command line client.)
>>>
>>> 2) what is the CPU load on the server and client while the tests are
>>> running.
>>> You can either run top with a short refresh time or turn the CPU
>>> monitoring flag
>>> on.  (I don't expect the CPU is overloaded, but let's find out).
>>>
>>> 3) the NDT server captures the analysis data during the
>>> Server-to-Client test so
>>> what does the more details page say?  (run the command line tool with
>>> the -ll
>>> option to extract this data or look at the web100srv.log file)
>>>
>>> However, looking at the code and the packet queuing message I think we
>>> have a
>>> probably answer.
>>>
>>> The packet queuing detected message is generated on the client during
>>> the
>>> analysis phase.  Both the client and the server measure the transfer
>>> speed
>>> (bytes transferred/time).  The client reports the value it
>>> calculated.  It is
>>> sent the speed the server calculated after the tests complete and the
>>> server
>>> sends this value as part of the larger test results back to the client
>>> for
>>> printing.  The formula is ((s2cspd-spdin)/s2cspd)*100 which is the
>>> servers
>>> calculation minus the clients calculation and converted to a
>>> percentage.  So
>>> your 80.16% says the server's calculation was 80% higher (a rough
>>> guess says the
>>> server calculated the speed around 7.4 Gbps).
>>>
>>> Given all this I'd look at the command line client's read data loop.
>>> It's not
>>> terminating properly and in this case ran about 50 seconds instead of
>>> 10.
>>>
>>> Try running the command line tool with some -d flags to print out
>>> debugging
>>> information.  That might tell you more about what's going on.
>>>
>>> Rich
>>>
>>> On 05/15/2013 04:25 PM, Byron Hicks wrote:
>>>> I'm reasonably certain that I have a clean path.
>>>>
>>>> Both boxes are running NDT, and I get the same result in both
>>>> directions:
>>>>
>>>> Houston:
>>>>
>>>> running 10s outbound test (client to server) . . . . .  9123.91 Mb/s
>>>> running 10s inbound test (server to client) . . . . . . 1434.11 Mb/s
>>>>
>>>> Dallas:
>>>>
>>>> running 10s outbound test (client to server) . . . . .  8953.05 Mb/s
>>>> running 10s inbound test (server to client) . . . . . . 1440.18 Mb/s
>>>>
>>>> If it were a traffic loss issue, I would expect that the
>>>> outbound/inbound numbers would flip, with the lower number being on the
>>>> "leg" of the duplex path that had the traffic loss.
>>>>
>>>> But I'm not.  Client to Server is 9Gb/s and Server to Client is
>>>> 1.4Gb/s,
>>>> regardless of which NDT server I'm testing from/to.  And considering
>>>> that I'm getting 9Gb/s on a 10Gb/s link using iperf in both directions,
>>>> I'm pretty sure packet loss is a not a factor.
>>>>
>>>> How do I interpret the following:
>>>>
>>>> Information [S2C]: Packet queuing detected: 80.16% (remote buffers)
>>>>
>>>> Where is the packet queuing happening?
>>>>
>>>>
>>>> On 05/15/2013 01:37 PM, Brian Tierney wrote:
>>>>> Another possibility is that I've seen cases where, on a path with
>>>>> packet
>>>>> loss, different clients seem to trigger different loss patterns.
>>>>>
>>>>> For example, here is on a clean path:
>>>>>
>>>>>
>>>>>> web100clt -n ps-lax-10g.cenic.net -b 33554432
>>>>> running 10s outbound test (client to server) . . . . .  2321.11 Mb/s
>>>>> running 10s inbound test (server to client) . . . . . . 2802.95 Mb/s
>>>>>
>>>>> vs bwctl:
>>>>>
>>>>>> bwctl -c ps-lax-10g.cenic.net -fm
>>>>> bwctl: Using tool: iperf
>>>>> [ 14] local 137.164.28.105 port 5001 connected with 198.129.254.98
>>>>> port 5001
>>>>> [ ID] Interval       Transfer     Bandwidth
>>>>> [ 14]  0.0-10.0 sec  2984 MBytes  2496 Mbits/sec
>>>>>
>>>>> performance is similar.
>>>>>
>>>>> ----------
>>>>>
>>>>> And here are the results for a path with packet loss:
>>>>>
>>>>>> web100clt -n ps-lax-10g.cenic.net -b 33554432
>>>>> running 10s outbound test (client to server) . . . . .  18.06 Mb/s
>>>>> running 10s inbound test (server to client) . . . . . . 2492.69 Mb/s
>>>>>
>>>>>> bwctl -c ps-lax-10g.cenic.net -fm
>>>>> [ 14] local 137.164.28.105 port 5001 connected with 198.129.254.150
>>>>> port 5001
>>>>> [ ID] Interval       Transfer     Bandwidth
>>>>> [ 14]  0.0-10.3 sec   552 MBytes   450 Mbits/sec
>>>>>
>>>>> Here iperf does 30x better than NDT (and btw, nuttcp results agree
>>>>> with the
>>>>> NDT results in this case)
>>>>>
>>>>> My guess is that different tools have different burst
>>>>> characteristics, and
>>>>> these trigger different amounts of packet loss.
>>>>>
>>>>
>>>
>>


-- 
Byron Hicks
Lonestar Education and Research Network
office: 972-883-4645
google: 972-746-2549
aim/skype: byronhicks