| Hey Byron,
Cheers, Aaron On May 20, 2013, at 9:03 PM, Byron Hicks <> wrote:
I'm using perfonar toolkit, so it is installed from the RPMS. If you send me a link, I'll install the RPMs and retest.
--
Byron Hicks
Lonestar Education and Research Networks
office: 972-746-2549
alternate: 972-883-4645
aim/skype: byronhicks
From: "Aaron Brown" <>
To:
Cc: "Richard Carlson" <>, "Nat Stoddard" <>,
Sent: Friday, May 17, 2013 3:42:12 PM
Subject: Re: Using NDT with 10 gigabit interfaces
Hey Byron,
Were you using the RPMs, or did you compile NDT on your own? If the former, I can get you some updated RPMs to test.
Cheers, Aaron On May 17, 2013, at 4:36 PM, Byron Hicks <> wrote: Rich,
Do you still need me to do more testing on my end, or was the
information provided by Nat Stoddard enough?
On 05/16/2013 10:39 PM, Richard Carlson wrote:
Nat;
This is good. As I thought, the CPU isn't the bottleneck.
The connection is sender limited, meaning its' not network or receiver
limited. The send/receive buffers don't limit the throughput.
The pkt-pair timing say OC-48 to 10 GE so that's good.
So lets look at the raw numbers.
CurMSS = 1448 (normal Ethernet frame)
Data Pkts Out = 7873701
Data Bytes Out = 2,147,483,647
Timesec = 10.00 sec
This is suspicious.
Bytes sent should be: 7,873,701 x 1448 = 11,401,119,048
Speed calculates to 9.1 Gbps So this is what the server acheived, but
not what is logged by the client.
But the server report 2.2 GB which converts to 1.7 Gbps. which is what
the spd: variable says.
I wonder if this is just a variable overflow error. The DataBytesOut
variable is a 32 bit counter. There is an HCDataOctetsOut variable (a
64 bit counter)
So, this is a problem that may impact some of the analysis messages.
line 1082 in web100srv.c sets the variable spd based on the DataBytesOut
value. Either the code should grab the HCDataOctetsOut value or a test
should be made and if s2cspd is greater that something like 5 Gbps, use
the HC value otherwise use the existing value.
I'm looking at the code and it show the server is able to right log
messages that might help here. Restart the server with 1 or 2 debug
flags (-d or -dd) and post the output here.
Got it !!!!!! While the DataByteOut vs HCDataOctetOut issue noted above
is a potential bug, it is not causing this bug.
The file test_s2c_clt.c is the culprit. The data read subroutine
declares the byte counter as an uint32_t. This is too small for the
amount of data we are getting at 10 Gbps, and it is overflowing.
Changing that to a uint64_t variable should fix this problem. The
server uses double, so it doesn't overflow.
So change line 32 of test_s2c_clt.c s/uint32_t/uint64_t
and recompile. See if that fixes this bug.
Same with the Java client, line 1253 of the Tcpbw100.java file declares
bytes as an int, this should be a double to prevent counter overflows at
10 Gbps.
Rich
On 05/16/2013 01:30 PM, Nat Stoddard wrote:
Hi Rich,
Thank you for taking notice of this.
1. I notice this behavior with both the web and the command line
clients. I only have the command line client to refer to when I am
testing between my two NDT servers. I get numbers very close to the
same in both directions:
server1 to server2 outbound 8517.57 Mb/s, inbound 2164.96 Mb/s
server2 to server1 outbound 9101.33 Mb/s, inbound 2245.74 Mb/s
2. Top reports the CPU load on the server as high as 49%. The client
goes up to 45%.
3. I have pasted the web100clt -ll output below:
$ web100clt -n lblnet-test.lbl.gov -ll
Testing network path for configuration and performance problems --
Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
checking for firewalls . . . . . . . . . . . . . . . . . . . Done
running 10s outbound test (client to server) . . . . . 9101.33 Mb/s
running 10s inbound test (server to client) . . . . . . 2245.74 Mb/s
The slowest link in the end-to-end path is a 2.4 Gbps OC-48 subnet
Information [S2C]: Packet queuing detected: 75.37% (remote buffers)
Server 'lblnet-test.lbl.gov' is not behind a firewall. [Connection to
the ephemeral port was successful]
Client is not behind a firewall. [Connection to the ephemeral port was
successful]
------ Web100 Detailed Analysis ------
Web100 reports the Round trip time = 10.54 msec;the Packet size = 1448
Bytes; and
No packet loss was observed.
This connection is receiver limited 1.67% of the time.
This connection is sender limited 98.30% of the time.
Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: ON
RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10
The theoretical network limit is 104855.13 Mbps
The NDT server has a 16384 KByte buffer which limits the throughput to
12148.82 Mbps
Your PC/Workstation has a 12269 KByte buffer which limits the throughput
to 9097.53 Mbps
The network based flow control limits the throughput to 13645.63 Mbps
Client Data reports link is ' 8', Client Acks report link is ' 9'
Server Data reports link is ' 9', Server Acks report link is ' 9'
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Client IP addresses are preserved End-to-End
CurMSS: 1448
X_Rcvbuf: 87380
X_Sndbuf: 16777216
AckPktsIn: 267378
AckPktsOut: 0
BytesRetrans: 0
CongAvoid: 0
CongestionOverCount: 0
CongestionSignals: 0
CountRTT: 267379
CurCwnd: 18844272
CurRTO: 210
CurRwinRcvd: 12521472
CurRwinSent: 6144
CurSsthresh: 2147483647
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 2147483647
DataPktsIn: 0
DataPktsOut: 7873701
DupAcksIn: 0
ECNEnabled: 0
FastRetran: 0
MaxCwnd: 18844272
MaxMSS: 1448
MaxRTO: 212
MaxRTT: 12
MaxRwinRcvd: 12563456
MaxRwinSent: 6144
MaxSsthresh: 0
MinMSS: 1448
MinRTO: 201
MinRTT: 0
MinRwinRcvd: 6144
MinRwinSent: 5792
NagleEnabled: 1
OtherReductions: 0
PktsIn: 267378
PktsOut: 7873701
PktsRetrans: 0
RcvWinScale: 10
SACKEnabled: 3
SACKsRcvd: 0
SendStall: 0
SlowStart: 13012
SampleRTT: 10
SmoothedRTT: 10
SndWinScale: 10
SndLimTimeRwin: 167989
SndLimTimeCwnd: 2531
SndLimTimeSender: 9878037
SndLimTransRwin: 4723
SndLimTransCwnd: 42
SndLimTransSender: 4766
SndLimBytesRwin: 352300576
SndLimBytesCwnd: 1710336
SndLimBytesSender: 2147483647
SubsequentTimeouts: 0
SumRTT: 2817061
Timeouts: 0
TimestampsEnabled: 1
WinScaleRcvd: 10
WinScaleSent: 10
DupAcksOut: 0
StartTimeUsec: 6541
Duration: 10050267
c2sData: 8
c2sAck: 9
s2cData: 9
s2cAck: 9
half_duplex: 0
link: 100
congestion: 0
bad_cable: 0
mismatch: 0
spd: 1709.69
bw: 104855.13
loss: 0.000000000
avgrtt: 10.54
waitsec: 0.00
timesec: 10.00
order: 0.0000
rwintime: 0.0167
sendtime: 0.9830
cwndtime: 0.0003
rwin: 95.8516
swin: 128.0000
cwin: 143.7704
rttsec: 0.010536
Sndbuf: 16777216
aspd: 0.00000
CWND-Limited: 96729.00
minCWNDpeak: -1
maxCWNDpeak: -1
CWNDpeaks: -1
On 5/15/13 7:33 PM, Richard Carlson wrote:
Byron;
As you and Nat have noted, the NDT server seems to have a problem with
10 Gbps
links. I have noticed this behavior before, but I do not have any 10
Gbps nodes
to play with.
If your willing to do some investigating, maybe we can figure out
what's going on.
The questions I would start with are:
1) are you using the web client or the command line client? (I think
it's the
command line client.)
2) what is the CPU load on the server and client while the tests are
running.
You can either run top with a short refresh time or turn the CPU
monitoring flag
on. (I don't expect the CPU is overloaded, but let's find out).
3) the NDT server captures the analysis data during the
Server-to-Client test so
what does the more details page say? (run the command line tool with
the -ll
option to extract this data or look at the web100srv.log file)
However, looking at the code and the packet queuing message I think we
have a
probably answer.
The packet queuing detected message is generated on the client during
the
analysis phase. Both the client and the server measure the transfer
speed
(bytes transferred/time). The client reports the value it
calculated. It is
sent the speed the server calculated after the tests complete and the
server
sends this value as part of the larger test results back to the client
for
printing. The formula is ((s2cspd-spdin)/s2cspd)*100 which is the
servers
calculation minus the clients calculation and converted to a
percentage. So
your 80.16% says the server's calculation was 80% higher (a rough
guess says the
server calculated the speed around 7.4 Gbps).
Given all this I'd look at the command line client's read data loop.
It's not
terminating properly and in this case ran about 50 seconds instead of
10.
Try running the command line tool with some -d flags to print out
debugging
information. That might tell you more about what's going on.
Rich
On 05/15/2013 04:25 PM, Byron Hicks wrote:
I'm reasonably certain that I have a clean path.
Both boxes are running NDT, and I get the same result in both
directions:
Houston:
running 10s outbound test (client to server) . . . . . 9123.91 Mb/s
running 10s inbound test (server to client) . . . . . . 1434.11 Mb/s
Dallas:
running 10s outbound test (client to server) . . . . . 8953.05 Mb/s
running 10s inbound test (server to client) . . . . . . 1440.18 Mb/s
If it were a traffic loss issue, I would expect that the
outbound/inbound numbers would flip, with the lower number being on the
"leg" of the duplex path that had the traffic loss.
But I'm not. Client to Server is 9Gb/s and Server to Client is
1.4Gb/s,
regardless of which NDT server I'm testing from/to. And considering
that I'm getting 9Gb/s on a 10Gb/s link using iperf in both directions,
I'm pretty sure packet loss is a not a factor.
How do I interpret the following:
Information [S2C]: Packet queuing detected: 80.16% (remote buffers)
Where is the packet queuing happening?
On 05/15/2013 01:37 PM, Brian Tierney wrote:
Another possibility is that I've seen cases where, on a path with
packet
loss, different clients seem to trigger different loss patterns.
For example, here is on a clean path:
web100clt -n ps-lax-10g.cenic.net -b 33554432
running 10s outbound test (client to server) . . . . . 2321.11 Mb/s
running 10s inbound test (server to client) . . . . . . 2802.95 Mb/s
vs bwctl:
bwctl -c ps-lax-10g.cenic.net -fm
bwctl: Using tool: iperf
[ 14] local 137.164.28.105 port 5001 connected with 198.129.254.98
port 5001
[ ID] Interval Transfer Bandwidth
[ 14] 0.0-10.0 sec 2984 MBytes 2496 Mbits/sec
performance is similar.
----------
And here are the results for a path with packet loss:
web100clt -n ps-lax-10g.cenic.net -b 33554432
running 10s outbound test (client to server) . . . . . 18.06 Mb/s
running 10s inbound test (server to client) . . . . . . 2492.69 Mb/s
bwctl -c ps-lax-10g.cenic.net -fm
[ 14] local 137.164.28.105 port 5001 connected with 198.129.254.150
port 5001
[ ID] Interval Transfer Bandwidth
[ 14] 0.0-10.3 sec 552 MBytes 450 Mbits/sec
Here iperf does 30x better than NDT (and btw, nuttcp results agree
with the
NDT results in this case)
My guess is that different tools have different burst
characteristics, and
these trigger different amounts of packet loss.
--
Byron Hicks
Lonestar Education and Research Network
office: 972-883-4645
google: 972-746-2549
aim/skype: byronhicks
|
