Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Help with inconsistent bwctl measurements

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Help with inconsistent bwctl measurements


Chronological Thread 
  • From: Eli Dart <>
  • To: Roderick Mooi <>, Alan Whinery <>,
  • Cc: perf-node-users <>
  • Subject: Re: [perfsonar-user] Help with inconsistent bwctl measurements
  • Date: Thu, 17 Oct 2013 10:36:51 -0700
  • Organization: Energy Sciences Network

Hi Roderick,

The "good" test looks like wire-speed 1Gbps to me.

The "bad" test could be due to loss on a short-latency path.

--eli



On 10/17/13 2:22 AM, Roderick Mooi wrote:
Hi Alan, Eli

I'm not seeing fluctuations in "good" or "bad" measurements.

Good:

RECEIVER START
bwctl: exec_line: iperf -B 196.21.48.249 -s -f m -m -p 5152 -t 20 -i 1
bwctl: start_tool: 3590989992.044877
------------------------------------------------------------
Server listening on TCP port 5152
Binding to local address 196.21.48.249
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ 14] local 196.21.48.249 port 5152 connected with 192.96.2.247 port 5152
[ ID] Interval Transfer Bandwidth
[ 14] 0.0- 1.0 sec 111 MBytes 929 Mbits/sec
[ 14] 1.0- 2.0 sec 112 MBytes 941 Mbits/sec
[ 14] 2.0- 3.0 sec 112 MBytes 942 Mbits/sec
[ 14] 3.0- 4.0 sec 112 MBytes 941 Mbits/sec
[ 14] 4.0- 5.0 sec 112 MBytes 941 Mbits/sec
[ 14] 5.0- 6.0 sec 112 MBytes 941 Mbits/sec
[ 14] 6.0- 7.0 sec 112 MBytes 941 Mbits/sec
[ 14] 7.0- 8.0 sec 112 MBytes 941 Mbits/sec
[ 14] 8.0- 9.0 sec 112 MBytes 941 Mbits/sec
[ 14] 9.0-10.0 sec 112 MBytes 941 Mbits/sec
[ 14] 10.0-11.0 sec 112 MBytes 942 Mbits/sec
[ 14] 11.0-12.0 sec 112 MBytes 941 Mbits/sec
[ 14] 12.0-13.0 sec 112 MBytes 941 Mbits/sec
[ 14] 13.0-14.0 sec 112 MBytes 941 Mbits/sec
[ 14] 14.0-15.0 sec 112 MBytes 942 Mbits/sec
[ 14] 15.0-16.0 sec 112 MBytes 941 Mbits/sec
[ 14] 16.0-17.0 sec 112 MBytes 941 Mbits/sec
[ 14] 17.0-18.0 sec 112 MBytes 941 Mbits/sec
[ 14] 18.0-19.0 sec 112 MBytes 941 Mbits/sec
[ 14] 19.0-20.0 sec 112 MBytes 941 Mbits/sec
[ 14] 0.0-20.5 sec 2298 MBytes 941 Mbits/sec
[ 14] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
bwctl: stop_exec: 3590990016.831918

RECEIVER END


Bad:

RECEIVER START
bwctl: exec_line: iperf -B 196.21.48.249 -s -f m -m -p 5149 -t 20 -i 1
bwctl: start_tool: 3590989696.322229
------------------------------------------------------------
Server listening on TCP port 5149
Binding to local address 196.21.48.249
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ 14] local 196.21.48.249 port 5149 connected with 192.96.2.247 port 5149
[ ID] Interval Transfer Bandwidth
[ 14] 0.0- 1.0 sec 12.8 MBytes 107 Mbits/sec
[ 14] 1.0- 2.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 14] 2.0- 3.0 sec 13.9 MBytes 116 Mbits/sec
[ 14] 3.0- 4.0 sec 18.1 MBytes 152 Mbits/sec
[ 14] 4.0- 5.0 sec 14.7 MBytes 124 Mbits/sec
[ 14] 5.0- 6.0 sec 16.1 MBytes 135 Mbits/sec
[ 14] 6.0- 7.0 sec 14.9 MBytes 125 Mbits/sec
[ 14] 7.0- 8.0 sec 10.3 MBytes 86.3 Mbits/sec
[ 14] 8.0- 9.0 sec 16.6 MBytes 139 Mbits/sec
[ 14] 9.0-10.0 sec 19.7 MBytes 165 Mbits/sec
[ 14] 10.0-11.0 sec 15.0 MBytes 126 Mbits/sec
[ 14] 11.0-12.0 sec 21.2 MBytes 178 Mbits/sec
[ 14] 12.0-13.0 sec 13.3 MBytes 112 Mbits/sec
[ 14] 13.0-14.0 sec 12.2 MBytes 102 Mbits/sec
[ 14] 14.0-15.0 sec 12.7 MBytes 107 Mbits/sec
[ 14] 15.0-16.0 sec 10.9 MBytes 91.2 Mbits/sec
[ 14] 16.0-17.0 sec 10.9 MBytes 91.6 Mbits/sec
[ 14] 17.0-18.0 sec 13.5 MBytes 114 Mbits/sec
[ 14] 18.0-19.0 sec 11.7 MBytes 97.8 Mbits/sec
[ 14] 19.0-20.0 sec 12.0 MBytes 100 Mbits/sec
[ 14] 0.0-20.1 sec 282 MBytes 118 Mbits/sec
[ 14] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
bwctl: stop_exec: 3590989721.229266

RECEIVER END

Referring to my complementary email to Brian and Ivan, do you have further
suggestions?

" I'm still puzzled by the fact that all tests between these 2 nodes and
others nodes on the network path are fine (i.e. I don't see this up-down
behaviour).
[see:
http://perfsonara.sanren.ac.za/serviceTest/index.cgi?eventType=bwctl
and
https://192.96.2.247/serviceTest/index.cgi?eventType=bwctl
]
"

Thanks!

Roderick

On 2013-10-16 at 19:07, Eli Dart
<>
wrote:


On 10/16/13 9:46 AM, Alan Whinery wrote:
You might also reveal something useful by using periodic reports in your
bwctl invocations (like " -i 1 ") you may find that the per second
reports show burstiness, or the lack of it.

I find this to be very very helpful.

There is a big difference between a clean ramp to a stable speed, and
wild fluctuation that is averaged.

A clean ramp to a stable speed argues against the presence of packet
loss. If performance is poor but stable, I would check the hosts and
the application, and then check for a clean bottleneck link. Wild
fluctuation points toward loss - check your router and switch buffers.
(And if "poor but stable" means fluctuating between 10Kbps and 30Kbps,
there is probably loss too :)

None of this is set in stone of course. However, I find that telling
bwctl to give periodic reports is very helpful indeed.

--eli



On 10/16/2013 6:26 AM, Wefel, Paul wrote:
Couple ideas

Run owamp between these two hosts looking for packet loss in only one
direction.
Check the switch interface that Dst is connected to looking for queue
drops and pause frames being sent.

I have also seen strange issues with some NICS when offloading is enabled
on them.

good luck, let us know what you find.

-paul
NCSA @ UIUC

-----Original Message-----
From: Roderick Mooi
<>
Date: Wednesday, October 16, 2013 5:07 AM
To:
""
<>,

""
<>
Subject: [perfsonar-user] Help with inconsistent bwctl measurements

Hi

I have been trying to locate the cause of inconsistent measurements
between two nodes for a few weeks now without success. The pattern I'm
seeing is available at:

https://192.96.2.247/serviceTest/bandwidthGraph.cgi?url=http://localhost:8
085/perfSONAR_PS/services/pSB&key=d9013ce7df20b8bbe45defeaeae785d6&keyR=0a
0ed6c928edf28976414a2cc7e87d6f&dstIP=192.96.2.247&srcIP=196.21.48.249&dst=
192.96.2.247&src=perfsonara.sanren.ac.za&type=TCP&length=2592000

Src-Dst is consistent but Dst-Src is not.

Manual tests (attached) show the same behaviour without any indication of
cause - measures 941 Mbps then drops to 189 Mbps (end) and back to 941
(nothing different in the logs between "good" measurements and "bad"
ones). The only time I've seen something similar is when I was testing
>from a 10 G interface to a 1 G interface which was subsequently being
flooded. In this case both interfaces are 1 G. I'm also not seeing any
problems with measurements along the path or between these nodes and any
other nodes. Additionally, there is very little (< 50 Mbps) real traffic
between these 2 nodes.

Any ideas?

Thanks!

Roderick

--
Roderick Mooi | SANREN Engineer
--

| +27 12 841 4111 | www.sanren.ac.za




--
This message is subject to the CSIR's copyright terms and conditions,
e-mail legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at
http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by
MailScanner,
and is believed to be clean.

Please consider the environment before printing this email.




--
Eli Dart, Network Engineer NOC: (510) 486-7600
ESnet Office of the CTO (AS293) (800) 333-7638
Lawrence Berkeley National Laboratory
PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3

--
This message is subject to the CSIR's copyright terms and conditions, e-mail
legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at
http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by
MailScanner,
and is believed to be clean.

Please consider the environment before printing this email.


--
Eli Dart, Network Engineer NOC: (510) 486-7600
ESnet Office of the CTO (AS293) (800) 333-7638
Lawrence Berkeley National Laboratory
PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3



Archive powered by MHonArc 2.6.16.

Top of Page