perfSONAR User Q&A and Other Discussion

[Eli Dart <>]

Hi Roderick,

The "good" test looks like wire-speed 1Gbps to me.

The "bad" test could be due to loss on a short-latency path.

                --eli



On 10/17/13 2:22 AM, Roderick Mooi wrote:
> Hi Alan, Eli
>
> I'm not seeing fluctuations in "good" or "bad" measurements.
>
> Good:
>
> RECEIVER START
> bwctl: exec_line: iperf -B 196.21.48.249 -s -f m -m -p 5152 -t 20 -i 1
> bwctl: start_tool: 3590989992.044877
> ------------------------------------------------------------
> Server listening on TCP port 5152
> Binding to local address 196.21.48.249
> TCP window size: 0.08 MByte (default)
> ------------------------------------------------------------
> [ 14] local 196.21.48.249 port 5152 connected with 192.96.2.247 port 5152
> [ ID] Interval       Transfer     Bandwidth
> [ 14]  0.0- 1.0 sec   111 MBytes   929 Mbits/sec
> [ 14]  1.0- 2.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  2.0- 3.0 sec   112 MBytes   942 Mbits/sec
> [ 14]  3.0- 4.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  4.0- 5.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  5.0- 6.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  6.0- 7.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  7.0- 8.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  8.0- 9.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  9.0-10.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 10.0-11.0 sec   112 MBytes   942 Mbits/sec
> [ 14] 11.0-12.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 12.0-13.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 13.0-14.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 14.0-15.0 sec   112 MBytes   942 Mbits/sec
> [ 14] 15.0-16.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 16.0-17.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 17.0-18.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 18.0-19.0 sec   112 MBytes   941 Mbits/sec
> [ 14] 19.0-20.0 sec   112 MBytes   941 Mbits/sec
> [ 14]  0.0-20.5 sec  2298 MBytes   941 Mbits/sec
> [ 14] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
> bwctl: stop_exec: 3590990016.831918
>
> RECEIVER END
>
>
> Bad:
>
> RECEIVER START
> bwctl: exec_line: iperf -B 196.21.48.249 -s -f m -m -p 5149 -t 20 -i 1
> bwctl: start_tool: 3590989696.322229
> ------------------------------------------------------------
> Server listening on TCP port 5149
> Binding to local address 196.21.48.249
> TCP window size: 0.08 MByte (default)
> ------------------------------------------------------------
> [ 14] local 196.21.48.249 port 5149 connected with 192.96.2.247 port 5149
> [ ID] Interval       Transfer     Bandwidth
> [ 14]  0.0- 1.0 sec  12.8 MBytes   107 Mbits/sec
> [ 14]  1.0- 2.0 sec  11.1 MBytes  93.3 Mbits/sec
> [ 14]  2.0- 3.0 sec  13.9 MBytes   116 Mbits/sec
> [ 14]  3.0- 4.0 sec  18.1 MBytes   152 Mbits/sec
> [ 14]  4.0- 5.0 sec  14.7 MBytes   124 Mbits/sec
> [ 14]  5.0- 6.0 sec  16.1 MBytes   135 Mbits/sec
> [ 14]  6.0- 7.0 sec  14.9 MBytes   125 Mbits/sec
> [ 14]  7.0- 8.0 sec  10.3 MBytes  86.3 Mbits/sec
> [ 14]  8.0- 9.0 sec  16.6 MBytes   139 Mbits/sec
> [ 14]  9.0-10.0 sec  19.7 MBytes   165 Mbits/sec
> [ 14] 10.0-11.0 sec  15.0 MBytes   126 Mbits/sec
> [ 14] 11.0-12.0 sec  21.2 MBytes   178 Mbits/sec
> [ 14] 12.0-13.0 sec  13.3 MBytes   112 Mbits/sec
> [ 14] 13.0-14.0 sec  12.2 MBytes   102 Mbits/sec
> [ 14] 14.0-15.0 sec  12.7 MBytes   107 Mbits/sec
> [ 14] 15.0-16.0 sec  10.9 MBytes  91.2 Mbits/sec
> [ 14] 16.0-17.0 sec  10.9 MBytes  91.6 Mbits/sec
> [ 14] 17.0-18.0 sec  13.5 MBytes   114 Mbits/sec
> [ 14] 18.0-19.0 sec  11.7 MBytes  97.8 Mbits/sec
> [ 14] 19.0-20.0 sec  12.0 MBytes   100 Mbits/sec
> [ 14]  0.0-20.1 sec   282 MBytes   118 Mbits/sec
> [ 14] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
> bwctl: stop_exec: 3590989721.229266
>
> RECEIVER END
>
> Referring to my complementary email to Brian and Ivan, do you have further 
> suggestions?
>
> " I'm still puzzled by the fact that all tests between these 2 nodes and 
> others nodes on the network path are fine (i.e. I don't see this up-down 
> behaviour).
> [see:
> http://perfsonara.sanren.ac.za/serviceTest/index.cgi?eventType=bwctl
> and
> https://192.96.2.247/serviceTest/index.cgi?eventType=bwctl
> ]
> "
>
> Thanks!
>
> Roderick
>
> > > > On 2013-10-16 at 19:07, Eli Dart 
> > > > <>
> > > >  wrote:
>
> > On 10/16/13 9:46 AM, Alan Whinery wrote:
> > > You might also reveal something useful by using periodic reports in your
> > > bwctl invocations (like " -i 1 ") you may find that the per second
> > > reports show burstiness, or the lack of it.
> >
> > I find this to be very very helpful.
> >
> > There is a big difference between a clean ramp to a stable speed, and
> > wild fluctuation that is averaged.
> >
> > A clean ramp to a stable speed argues against the presence of packet
> > loss.  If performance is poor but stable, I would check the hosts and
> > the application, and then check for a clean bottleneck link.  Wild
> > fluctuation points toward loss - check your router and switch buffers.
> > (And if "poor but stable" means fluctuating between 10Kbps and 30Kbps,
> > there is probably loss too :)
> >
> > None of this is set in stone of course.  However, I find that telling
> > bwctl to give periodic reports is very helpful indeed.
> >
> >                 --eli
> >
> >
> > > On 10/16/2013 6:26 AM, Wefel, Paul wrote:
> > > > Couple ideas
> > > >
> > > > Run owamp between these two hosts looking for packet loss in only one
> > > > direction.
> > > > Check the switch interface that Dst is connected to looking for queue
> > > > drops and pause frames being sent.
> > > >
> > > > I have also seen strange issues with some NICS when offloading is enabled
> > > > on them.
> > > >
> > > > good luck, let us know what you find.
> > > >
> > > > -paul
> > > > NCSA @ UIUC
> > > >
> > > > -----Original Message-----
> > > > From: Roderick Mooi 
> > > > <>
> > > > Date: Wednesday, October 16, 2013 5:07 AM
> > > > To: 
> > > > ""
> > > > <>,
> > > >  
> > > > ""
> > > > <>
> > > > Subject: [perfsonar-user] Help with inconsistent bwctl measurements
> > > >
> > > > > Hi
> > > > >
> > > > > I have been trying to locate the cause of inconsistent measurements
> > > > > between two nodes for a few weeks now without success. The pattern I'm
> > > > > seeing is available at:
> > > > >
> > > > > https://192.96.2.247/serviceTest/bandwidthGraph.cgi?url=http://localhost:8
> > > > > 085/perfSONAR_PS/services/pSB&key=d9013ce7df20b8bbe45defeaeae785d6&keyR=0a
> > > > > 0ed6c928edf28976414a2cc7e87d6f&dstIP=192.96.2.247&srcIP=196.21.48.249&dst=
> > > > > 192.96.2.247&src=perfsonara.sanren.ac.za&type=TCP&length=2592000
> > > > >
> > > > > Src-Dst is consistent but Dst-Src is not.
> > > > >
> > > > > Manual tests (attached) show the same behaviour without any indication of
> > > > > cause - measures 941 Mbps then drops to 189 Mbps (end) and back to 941
> > > > > (nothing different in the logs between "good" measurements and "bad"
> > > > > ones). The only time I've seen something similar is when I was testing
> > > > >from a 10 G interface to a 1 G interface which was subsequently being
> > > > > flooded. In this case both interfaces are 1 G. I'm also not seeing any
> > > > > problems with measurements along the path or between these nodes and any
> > > > > other nodes. Additionally, there is very little (< 50 Mbps) real traffic
> > > > > between these 2 nodes.
> > > > >
> > > > > Any ideas?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Roderick
> > > > >
> > > > > --
> > > > > Roderick Mooi      | SANREN Engineer
> > > > > --
> > > > >
> > > > >  | +27 12 841 4111 | www.sanren.ac.za
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > This message is subject to the CSIR's copyright terms and conditions,
> > > > > e-mail legal notice, and implemented Open Document Format (ODF) standard.
> > > > > The full disclaimer details can be found at
> > > > > http://www.csir.co.za/disclaimer.html.
> > > > >
> > > > > This message has been scanned for viruses and dangerous content by
> > > > > MailScanner,
> > > > > and is believed to be clean.
> > > > >
> > > > > Please consider the environment before printing this email.
> >
> > --
> > Eli Dart, Network Engineer                          NOC: (510) 486-7600
> > ESnet Office of the CTO (AS293)                          (800) 333-7638
> > Lawrence Berkeley National Laboratory
> > PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3
> >
> > --
> > This message is subject to the CSIR's copyright terms and conditions, e-mail
> > legal notice, and implemented Open Document Format (ODF) standard.
> > The full disclaimer details can be found at
> > http://www.csir.co.za/disclaimer.html.
> >
> > This message has been scanned for viruses and dangerous content by
> > MailScanner,
> > and is believed to be clean.
> >
> > Please consider the environment before printing this email.

--
Eli Dart, Network Engineer                          NOC: (510) 486-7600
ESnet Office of the CTO (AS293)                          (800) 333-7638
Lawrence Berkeley National Laboratory
PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3