perfSONAR User Q&A and Other Discussion

["Roderick Mooi" <>]

Hi Alan, Eli

I'm not seeing fluctuations in "good" or "bad" measurements.

Good:

RECEIVER START
bwctl: exec_line: iperf -B 196.21.48.249 -s -f m -m -p 5152 -t 20 -i 1
bwctl: start_tool: 3590989992.044877
------------------------------------------------------------
Server listening on TCP port 5152
Binding to local address 196.21.48.249
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ 14] local 196.21.48.249 port 5152 connected with 192.96.2.247 port 5152
[ ID] Interval       Transfer     Bandwidth
[ 14]  0.0- 1.0 sec   111 MBytes   929 Mbits/sec
[ 14]  1.0- 2.0 sec   112 MBytes   941 Mbits/sec
[ 14]  2.0- 3.0 sec   112 MBytes   942 Mbits/sec
[ 14]  3.0- 4.0 sec   112 MBytes   941 Mbits/sec
[ 14]  4.0- 5.0 sec   112 MBytes   941 Mbits/sec
[ 14]  5.0- 6.0 sec   112 MBytes   941 Mbits/sec
[ 14]  6.0- 7.0 sec   112 MBytes   941 Mbits/sec
[ 14]  7.0- 8.0 sec   112 MBytes   941 Mbits/sec
[ 14]  8.0- 9.0 sec   112 MBytes   941 Mbits/sec
[ 14]  9.0-10.0 sec   112 MBytes   941 Mbits/sec
[ 14] 10.0-11.0 sec   112 MBytes   942 Mbits/sec
[ 14] 11.0-12.0 sec   112 MBytes   941 Mbits/sec
[ 14] 12.0-13.0 sec   112 MBytes   941 Mbits/sec
[ 14] 13.0-14.0 sec   112 MBytes   941 Mbits/sec
[ 14] 14.0-15.0 sec   112 MBytes   942 Mbits/sec
[ 14] 15.0-16.0 sec   112 MBytes   941 Mbits/sec
[ 14] 16.0-17.0 sec   112 MBytes   941 Mbits/sec
[ 14] 17.0-18.0 sec   112 MBytes   941 Mbits/sec
[ 14] 18.0-19.0 sec   112 MBytes   941 Mbits/sec
[ 14] 19.0-20.0 sec   112 MBytes   941 Mbits/sec
[ 14]  0.0-20.5 sec  2298 MBytes   941 Mbits/sec
[ 14] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
bwctl: stop_exec: 3590990016.831918

RECEIVER END


Bad:

RECEIVER START
bwctl: exec_line: iperf -B 196.21.48.249 -s -f m -m -p 5149 -t 20 -i 1
bwctl: start_tool: 3590989696.322229
------------------------------------------------------------
Server listening on TCP port 5149
Binding to local address 196.21.48.249
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ 14] local 196.21.48.249 port 5149 connected with 192.96.2.247 port 5149
[ ID] Interval       Transfer     Bandwidth
[ 14]  0.0- 1.0 sec  12.8 MBytes   107 Mbits/sec
[ 14]  1.0- 2.0 sec  11.1 MBytes  93.3 Mbits/sec
[ 14]  2.0- 3.0 sec  13.9 MBytes   116 Mbits/sec
[ 14]  3.0- 4.0 sec  18.1 MBytes   152 Mbits/sec
[ 14]  4.0- 5.0 sec  14.7 MBytes   124 Mbits/sec
[ 14]  5.0- 6.0 sec  16.1 MBytes   135 Mbits/sec
[ 14]  6.0- 7.0 sec  14.9 MBytes   125 Mbits/sec
[ 14]  7.0- 8.0 sec  10.3 MBytes  86.3 Mbits/sec
[ 14]  8.0- 9.0 sec  16.6 MBytes   139 Mbits/sec
[ 14]  9.0-10.0 sec  19.7 MBytes   165 Mbits/sec
[ 14] 10.0-11.0 sec  15.0 MBytes   126 Mbits/sec
[ 14] 11.0-12.0 sec  21.2 MBytes   178 Mbits/sec
[ 14] 12.0-13.0 sec  13.3 MBytes   112 Mbits/sec
[ 14] 13.0-14.0 sec  12.2 MBytes   102 Mbits/sec
[ 14] 14.0-15.0 sec  12.7 MBytes   107 Mbits/sec
[ 14] 15.0-16.0 sec  10.9 MBytes  91.2 Mbits/sec
[ 14] 16.0-17.0 sec  10.9 MBytes  91.6 Mbits/sec
[ 14] 17.0-18.0 sec  13.5 MBytes   114 Mbits/sec
[ 14] 18.0-19.0 sec  11.7 MBytes  97.8 Mbits/sec
[ 14] 19.0-20.0 sec  12.0 MBytes   100 Mbits/sec
[ 14]  0.0-20.1 sec   282 MBytes   118 Mbits/sec
[ 14] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
bwctl: stop_exec: 3590989721.229266

RECEIVER END

Referring to my complementary email to Brian and Ivan, do you have further 
suggestions?

" I'm still puzzled by the fact that all tests between these 2 nodes and 
others nodes on the network path are fine (i.e. I don't see this up-down 
behaviour).
[see:
http://perfsonara.sanren.ac.za/serviceTest/index.cgi?eventType=bwctl 
and
https://192.96.2.247/serviceTest/index.cgi?eventType=bwctl 
]
"

Thanks!

Roderick

>>> On 2013-10-16 at 19:07, Eli Dart 
>>> <>
>>>  wrote:

> 
> On 10/16/13 9:46 AM, Alan Whinery wrote:
>> You might also reveal something useful by using periodic reports in your
>> bwctl invocations (like " -i 1 ") you may find that the per second
>> reports show burstiness, or the lack of it.
> 
> I find this to be very very helpful.
> 
> There is a big difference between a clean ramp to a stable speed, and 
> wild fluctuation that is averaged.
> 
> A clean ramp to a stable speed argues against the presence of packet 
> loss.  If performance is poor but stable, I would check the hosts and 
> the application, and then check for a clean bottleneck link.  Wild 
> fluctuation points toward loss - check your router and switch buffers. 
> (And if "poor but stable" means fluctuating between 10Kbps and 30Kbps, 
> there is probably loss too :)
> 
> None of this is set in stone of course.  However, I find that telling 
> bwctl to give periodic reports is very helpful indeed.
> 
>               --eli
> 
> 
>>
>> On 10/16/2013 6:26 AM, Wefel, Paul wrote:
>>> Couple ideas
>>>
>>> Run owamp between these two hosts looking for packet loss in only one
>>> direction.
>>> Check the switch interface that Dst is connected to looking for queue
>>> drops and pause frames being sent.
>>>
>>> I have also seen strange issues with some NICS when offloading is enabled
>>> on them.
>>>
>>> good luck, let us know what you find.
>>>
>>> -paul
>>> NCSA @ UIUC
>>>
>>> -----Original Message-----
>>> From: Roderick Mooi 
>>> <>
>>> Date: Wednesday, October 16, 2013 5:07 AM
>>> To: 
>>> ""
>>> <>,
>>>  
>>> ""
>>> <>
>>> Subject: [perfsonar-user] Help with inconsistent bwctl measurements
>>>
>>>> Hi
>>>>
>>>> I have been trying to locate the cause of inconsistent measurements
>>>> between two nodes for a few weeks now without success. The pattern I'm
>>>> seeing is available at:
>>>>
>>>> https://192.96.2.247/serviceTest/bandwidthGraph.cgi?url=http://localhost:8
>>>>  
>>>> 085/perfSONAR_PS/services/pSB&key=d9013ce7df20b8bbe45defeaeae785d6&keyR=0a
>>>> 0ed6c928edf28976414a2cc7e87d6f&dstIP=192.96.2.247&srcIP=196.21.48.249&dst=
>>>> 192.96.2.247&src=perfsonara.sanren.ac.za&type=TCP&length=2592000
>>>>
>>>> Src-Dst is consistent but Dst-Src is not.
>>>>
>>>> Manual tests (attached) show the same behaviour without any indication of
>>>> cause - measures 941 Mbps then drops to 189 Mbps (end) and back to 941
>>>> (nothing different in the logs between "good" measurements and "bad"
>>>> ones). The only time I've seen something similar is when I was testing
>>> >from a 10 G interface to a 1 G interface which was subsequently being
>>>> flooded. In this case both interfaces are 1 G. I'm also not seeing any
>>>> problems with measurements along the path or between these nodes and any
>>>> other nodes. Additionally, there is very little (< 50 Mbps) real traffic
>>>> between these 2 nodes.
>>>>
>>>> Any ideas?
>>>>
>>>> Thanks!
>>>>
>>>> Roderick
>>>>
>>>> --
>>>> Roderick Mooi      | SANREN Engineer
>>>> --
>>>> 
>>>>  | +27 12 841 4111 | www.sanren.ac.za 
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> This message is subject to the CSIR's copyright terms and conditions,
>>>> e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>>> The full disclaimer details can be found at
>>>> http://www.csir.co.za/disclaimer.html.
>>>>
>>>> This message has been scanned for viruses and dangerous content by
>>>> MailScanner,
>>>> and is believed to be clean.
>>>>
>>>> Please consider the environment before printing this email.
>>>>
>>>
>>
> 
> -- 
> Eli Dart, Network Engineer                          NOC: (510) 486-7600
> ESnet Office of the CTO (AS293)                          (800) 333-7638
> Lawrence Berkeley National Laboratory
> PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3
> 
> -- 
> This message is subject to the CSIR's copyright terms and conditions, 
> e-mail 
> legal notice, and implemented Open Document Format (ODF) standard. 
> The full disclaimer details can be found at 
> http://www.csir.co.za/disclaimer.html.
> 
> This message has been scanned for viruses and dangerous content by 
> MailScanner, 
> and is believed to be clean.
> 
> Please consider the environment before printing this email.

-- 
This message is subject to the CSIR's copyright terms and conditions, e-mail 
legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at 
http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by 
MailScanner, 
and is believed to be clean.

Please consider the environment before printing this email.