perfSONAR User Q&A and Other Discussion

["Koch, Brian" <>]

Sorry for late response. Here’s the results:

 

[root@sys-its-70vlk81 ~]# web100clt -ll -n 129.120.255.107

Testing network path for configuration and performance problems  --  Using IPv4 address

Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done

checking for firewalls . . . . . . . . . . . . . . . . . . .  Failed to connect: No route to host

No sockets could be created that match requirements.

Done

running 10s outbound test (client to server) . . . . .  1034.25 Mb/s

running 10s inbound test (server to client) . . . . . . 1145.69 Mb/s

sending meta information to server . . . . . Done

The slowest link in the end-to-end path is a 2.4 Gbps OC-48 subnet

Information: Other network traffic is congesting the link

Server '129.120.255.107' is probably behind a firewall. [Connection to the ephemeral port failed]

Client is probably behind a firewall. [Connection to the ephemeral port failed]

 

        ------  Web100 Detailed Analysis  ------

 

Web100 reports the Round trip time = 6.68 msec;the Packet size = 1448 Bytes; and

There were 2080 packets retransmitted, 6471 duplicate acks received, and 9698 SACK blocks received

Packets arrived out-of-order 1.40% of the time.

This connection is sender limited 96.88% of the time.

This connection is network limited 3.02% of the time.

 

    Web100 reports TCP negotiated the optional Performance Settings to:

RFC 2018 Selective Acknowledgment: ON

RFC 896 Nagle Algorithm: ON

RFC 3168 Explicit Congestion Notification: OFF

RFC 1323 Time Stamping: ON

RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10

The theoretical network limit is 496.67 Mbps

The NDT server has a 7166 KByte buffer which limits the throughput to 8377.02 Mbps

Your PC/Workstation has a 3604 KByte buffer which limits the throughput to 4213.11 Mbps

The network based flow control limits the throughput to 3063.12 Mbps

 

Client Data reports link is '  8', Client Acks report link is '  8'

Server Data reports link is '  9', Server Acks report link is '  9'

Packet size is preserved End-to-End

Information: Network Address Translation (NAT) box is modifying the Server's IP address

        Server says [::ffff:129.120.255.107] but Client says [ 129.120.255.107]

Information: Network Address Translation (NAT) box is modifying the Client's IP address

        Server says [::ffff:10.3.254.5] but Client says [ 10.3.254.5]

CurMSS: 1448

X_Rcvbuf: 87380

X_Sndbuf: 7337880

AckPktsIn: 462646

AckPktsOut: 0

BytesRetrans: 3011840

CongAvoid: 448813

CongestionOverCount: 2975

CongestionSignals: 11

CountRTT: 455842

CurCwnd: 912240

CurRTO: 206

CurRwinRcvd: 3690496

CurRwinSent: 15360

CurSsthresh: 525624

DSACKDups: 0

DataBytesIn: 0

DataBytesOut: 1438758968

DataPktsIn: 0

DataPktsOut: 992907

DupAcksIn: 6471

ECNEnabled: 0

FastRetran: 11

MaxCwnd: 2683144

MaxMSS: 1448

MaxRTO: 214

MaxRTT: 17

MaxRwinRcvd: 3690496

MaxRwinSent: 15360

MaxSsthresh: 1342296

MinMSS: 1428

MinRTO: 201

MinRTT: 0

MinRwinRcvd: 15360

MinRwinSent: 14480

NagleEnabled: 1

OtherReductions: 1

PktsIn: 462646

PktsOut: 992907

PktsRetrans: 2080

RcvWinScale: 10

SACKEnabled: 3

SACKsRcvd: 9698

SendStall: 0

SlowStart: 2154

SampleRTT: 6

SmoothedRTT: 6

SndWinScale: 10

SndLimTimeRwin: 9995

SndLimTimeCwnd: 312629

SndLimTimeSender: 10024907

SndLimTransRwin: 24

SndLimTransCwnd: 10662

SndLimTransSender: 10685

SndLimBytesRwin: 2158288

SndLimBytesCwnd: 676065576

SndLimBytesSender: 760535104

SubsequentTimeouts: 0

SumRTT: 3046221

Timeouts: 0

TimestampsEnabled: 1

WinScaleRcvd: 10

WinScaleSent: 10

DupAcksOut: 0

StartTimeUsec: 151905

Duration: 10347619

c2sData: 8

c2sAck: 8

s2cData: 9

s2cAck: 9

half_duplex: 0

link: 0

congestion: 1

bad_cable: 0

mismatch: 0

spd: 1112.35

bw: 496.67

loss: 0.000011079

avgrtt: 6.68

waitsec: 0.00

timesec: 10.00

order: 0.0140

rwintime: 0.0010

sendtime: 0.9688

cwndtime: 0.0302

rwin: 28.1562

swin: 55.9836

cwin: 20.4708

rttsec: 0.006683

Sndbuf: 7337880

aspd: 0.00000

CWND-Limited: 156535.00

minCWNDpeak: 445984

maxCWNDpeak: 2683144

CWNDpeaks: 12

 

From: Alan Whinery [mailto:]
Sent: Monday, May 05, 2014 1:28 PM
To: Koch, Brian;
Cc:
Subject: RE: [perfsonar-user] bwctl speed questions

 

Interesting. A lot of intrigue there -- SACKs, out-of-orders, etc.

Could you do one more with

web100clt -ll -n <target>
?

I should have suggested multiple L but it's been a while since I had looked at the invocation spec. This will include a dump of all the web100 variables, like:

CurMSS: 1448
X_Rcvbuf: 87380
X_Sndbuf: 542520
AckPktsIn: 42069
AckPktsOut: 0
BytesRetrans: 97016
CongAvoid: 40313
CongestionOverCount: 77
CongestionSignals: 16
CountRTT: 40601
CurCwnd: 115840
CurRTO: 209
CurRwinRcvd: 397312
CurRwinSent: 18432
CurSsthresh: 60816
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 117766944
DataPktsIn: 0
DataPktsOut: 81047
DupAcksIn: 1469
ECNEnabled: 0
FastRetran: 16
MaxCwnd: 198376
MaxMSS: 1448
MaxRTO: 214
MaxRTT: 16
MaxRwinRcvd: 397312
MaxRwinSent: 18432
MaxSsthresh: 92672
MinMSS: 1448
MinRTO: 201
MinRTT: 1
MinRwinRcvd: 15360
MinRwinSent: 17896
NagleEnabled: 1
OtherReductions: 29
PktsIn: 42069
PktsOut: 81047
PktsRetrans: 67
RcvWinScale: 10
SACKEnabled: 3
SACKsRcvd: 1705
SendStall: 0
SlowStart: 142
SampleRTT: 9
SmoothedRTT: 9
SndWinScale: 10
SndLimTimeRwin: 16984
SndLimTimeCwnd: 1737301
SndLimTimeSender: 8634977
SndLimTransRwin: 11
SndLimTransCwnd: 6393
SndLimTransSender: 6400
SndLimBytesRwin: 253616
SndLimBytesCwnd: 69329000
SndLimBytesSender: 48184328
SubsequentTimeouts: 0
SumRTT: 316680
Timeouts: 0
TimestampsEnabled: 1
WinScaleRcvd: 10
WinScaleSent: 10
DupAcksOut: 0
StartTimeUsec: 299246
Duration: 10389356
c2sData: 5
c2sAck: 5
s2cData: 5
s2cAck: 5
half_duplex: 0
link: 100
congestion: 1
bad_cable: 0
mismatch: 0
spd: 90.68
bw: 100.81
loss: 0.000197416
avgrtt: 7.80
waitsec: 0.00
timesec: 10.00
order: 0.0349
rwintime: 0.0016
sendtime: 0.8311
cwndtime: 0.1672
rwin: 3.0312
swin: 4.1391
cwin: 1.5135
rttsec: 0.007800
Sndbuf: 542520
aspd: 0.00000
CWND-Limited: 23959.00
minCWNDpeak: 52128
maxCWNDpeak: 179552
CWNDpeaks: 30

On May 5, 2014 6:07:56 AM HST, "Koch, Brian" wrote:

sys-its-70vlk81 ~]# web100clt -l -n 129.120.255.107

Testing network path for configuration and performance problems  --  Using IPv4 address

Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done

checking for firewalls . . . . . . . . . . . . . . . . . . .  Failed to connect: No route to host

No sockets could be created that match requirements.

Done

running 10s outbound test (client to server) . . . . .  1049.25 Mb/s

running 10s inbound test (server to client) . . . . . . 1132.42 Mb/s

sending meta information to server . . . . . Done

The slowest link in the end-to-end path is a 2.4 Gbps OC-48 subnet

Information: Other network traffic is congesting the link

Server '129.120.255.107' is probably behind a firewall. [Connection to the ephemeral port failed]

Client is probably behind a firewall. [Connection to the ephemeral port failed]

        ------  Web100 Detailed Analysis  ------

Web100 reports the Round trip time = 9.06 msec;the Packet size = 1448 Bytes; and

There were 1781 packets retransmitted, 6271 duplicate acks received, and 10121 SACK blocks received

Packets arrived out-of-order 1.36% of the time.

This connection is sender limited 96.83% of the time.

This connection is network limited 3.08% of the time.

    Web100 reports TCP negotiated the optional Performance Settings to:

RFC 2018 Selective Acknowledgment: ON

RFC 896 Nagle Algorithm: ON

RFC 3168 Explicit Congestion Notification: OFF

RFC 1323 Time Stamping: ON

RFC 1323 Window Scaling: ON; Scaling Factors - Server=10, Client=10

The theoretical network limit is 311.75 Mbps

The NDT server has a 5971 KByte buffer which limits the throughput to 5149.34 Mbps

Your PC/Workstation has a 3770 KByte buffer which limits the throughput to 3251.25 Mbps

The network based flow control limits the throughput to 1884.12 Mbps

Client Data reports link is '  8', Client Acks report link is '  8'

Server Data reports link is '  9', Server Acks report link is '  9'

Packet size is preserved End-to-End

Information: Network Address Translation (NAT) box is modifying the Server's IP address

        Server says [::ffff:129.120.255.107] but Client says [ 129.120.255.107]

Information: Network Address Translation (NAT) box is modifying the Client's IP address

        Server says [::ffff:10.3.254.5] but Client says [ 10.3.254.5]

-----Original Message-----

From:  [] On Behalf Of Alan Whinery

Sent: Friday, May 02, 2014 4:10 PM

To: 

Subject: Re: [perfsonar-user] bwctl speed questions

Brian  T. - Yes, he said there was a firewall.

Brian Koch,

If NDT is enabled as a service on the target host, can you do

web100clt -l -n <targethost>

and send the results?

Note: if you are sanitizing, the IP target's name or IP appears once in the output.

On 5/2/2014 11:04 AM, Brian Tierney wrote:

 With 909 retransmits, you will never get much about 1Gbps.

 How "full" is that path? Is there a firewall on the path? Also look at 

 errors on all your router / switch interfaces.

 There may be a dirty fiber.

 On Fri, May 2, 2014 at 1:52 PM, Koch, Brian  wrote:

 Latency shows ~0.7ms with 0% loss.

 -----Original Message-----

 From: Brian Tierney []

 Sent: Friday, May 02, 2014 3:35 PM

 Cc: Koch, Brian; 

 Subject: Re: [perfsonar-user] bwctl speed questions

 Brian: What latency is the path you are testing? Does owamp show any packet loss on that path?

 Try doing a test with iperf3, which will give you TCP retransmit  count.

 bwctl -c hostname -I iperf3 -i1 -fm -x -t20

 On Fri, May 2, 2014 at 7:47 AM, Aaron Brown  wrote:

 Hey Brian,

 Unfortunately, there’s not a way to do multiple streams currently. The idea was to show how single stream performance looked since any network can be made to look good with enough streams :) The next version of the Toolkit should include the ability to configure number of streams, however.

 Cheers,

 Aaron

 On May 2, 2014, at 10:04 AM,  wrote:

 Is there a way to run multiple data streams? I'm not sure why the 

 data path is fully tested.

 Connections:

 Perfsonar --> core router (10G)

 core router --> firewall (10G) (fw throughput = 2G) firewall --> 

 outside WAN router (10G) outside WAN router --> perfsonar (10G)

 Theoretically, I should be seeing close to 2Gbps on my bwctl 

 graphs, but graphs are barely showing over 1Gbps.

 Are there some settings I need to tweak? I'm a little new to this.

 --

 Brian Tierney, http://www.es.net/tierney

 Energy Sciences Network (ESnet), Berkeley National Lab

 http://fasterdata.es.net