perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

As Matt noted, you will want to open a ticket with a coordination team or a 
NOC if you think there is a real problem that needs to be examined.  This 
list is more for the support of the tools, and not necessarily for the the 
debugging of problems.  

A level setting statement regarding the regular testing graphs, is that BWCTL 
is configured to run a single stream of TCP for 20 seconds, meaning the 
plotted number you see is a fragile representation of the network at that 
point in time.  Unless you spend a lot of time tuning the machines and the 
path, it is unlikely that you would see extremely high (e.g. 9Gbps + numbers) 
for regular testing with the tool defaults.  Running the tool by hand, and 
tweaking the time of the test, the size of the window, or the number of 
streams, may produce something better.  Maybe give that a try to see what is 
really going on, a side effect of doing a by hand test is that you will see 
things like retransmissions,  For example, here are tests between your two 
facilities featuring a lot of retransmissions, normally a sign that there is 
something in the path (congestion, lack of buffering, packet disruption 
devices, etc.) that is impacting TCP:

> [zurawski@bnl-pt1
>  ~]$ bwctl -T iperf3 -f m -t 30 -i 1 -c perfsonar-cms2.itns.purdue.edu -s 
> perfsonar.ultralight.org -L 1000
> bwctl: Using tool: iperf3
> bwctl: 50 seconds until test results available
> 
> SENDER START
> Connecting to host 128.211.143.4, port 5018
> [ 15] local 192.84.86.121 port 60673 connected to 128.211.143.4 port 5018
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [ 15]   0.00-1.00   sec  32.5 MBytes   273 Mbits/sec   42   1.78 MBytes     
>   
> [ 15]   1.00-2.00   sec  27.5 MBytes   231 Mbits/sec    0   1.90 MBytes     
>   
> [ 15]   2.00-3.00   sec  31.2 MBytes   262 Mbits/sec    0   2.41 MBytes     
>   
> [ 15]   3.00-4.00   sec  38.8 MBytes   325 Mbits/sec   10   1.70 MBytes     
>   
> [ 15]   4.00-5.00   sec  28.8 MBytes   241 Mbits/sec    0   1.84 MBytes     
>   
> [ 15]   5.00-6.00   sec  30.0 MBytes   252 Mbits/sec    0   2.39 MBytes     
>   
> [ 15]   6.00-7.00   sec  43.8 MBytes   367 Mbits/sec    0   3.72 MBytes     
>   
> [ 15]   7.00-8.00   sec  70.0 MBytes   587 Mbits/sec    0   5.88 MBytes     
>   
> [ 15]   8.00-9.00   sec  70.0 MBytes   587 Mbits/sec    1   3.45 MBytes     
>   
> [ 15]   9.00-10.00  sec  52.5 MBytes   440 Mbits/sec    0   3.68 MBytes     
>   
> [ 15]  10.00-11.00  sec  61.2 MBytes   514 Mbits/sec    0   4.64 MBytes     
>   
> [ 15]  11.00-12.00  sec  80.0 MBytes   671 Mbits/sec    0   6.41 MBytes     
>   
> [ 15]  12.00-13.00  sec   120 MBytes  1007 Mbits/sec    0   9.17 MBytes     
>   
> [ 15]  13.00-14.00  sec   159 MBytes  1332 Mbits/sec    0   11.3 MBytes     
>   
> [ 15]  14.00-15.00  sec  66.2 MBytes   556 Mbits/sec   21   2.61 MBytes     
>   
> [ 15]  15.00-16.00  sec  40.0 MBytes   336 Mbits/sec    0   2.83 MBytes     
>   
> [ 15]  16.00-17.00  sec  51.2 MBytes   430 Mbits/sec    0   3.80 MBytes     
>   
> [ 15]  17.00-18.00  sec  67.5 MBytes   566 Mbits/sec    0   5.55 MBytes     
>   
> [ 15]  18.00-19.00  sec   100 MBytes   839 Mbits/sec    0   8.16 MBytes     
>   
> [ 15]  19.00-20.00  sec   144 MBytes  1206 Mbits/sec    0   11.1 MBytes     
>   
> [ 15]  20.00-21.00  sec   168 MBytes  1405 Mbits/sec    0   11.3 MBytes     
>   
> [ 15]  21.00-22.00  sec   169 MBytes  1416 Mbits/sec    0   11.3 MBytes     
>   
> [ 15]  22.00-23.00  sec   169 MBytes  1416 Mbits/sec    0   11.4 MBytes     
>   
> [ 15]  23.00-24.00  sec   169 MBytes  1416 Mbits/sec    0   11.4 MBytes     
>   
> [ 15]  24.00-25.00  sec   169 MBytes  1416 Mbits/sec    0   11.4 MBytes     
>   
> [ 15]  25.00-26.00  sec   169 MBytes  1416 Mbits/sec    0   11.4 MBytes     
>   
> [ 15]  26.00-27.00  sec   169 MBytes  1416 Mbits/sec    0   11.4 MBytes     
>   
> [ 15]  27.00-28.00  sec   169 MBytes  1416 Mbits/sec    0   11.4 MBytes     
>   
> [ 15]  28.00-29.00  sec   169 MBytes  1416 Mbits/sec    0   11.4 MBytes     
>   
> [ 15]  29.00-30.00  sec   169 MBytes  1416 Mbits/sec    0   11.5 MBytes     
>   
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [ 15]   0.00-30.00  sec  2.93 GBytes   839 Mbits/sec   74             sender
> [ 15]   0.00-30.00  sec  2.93 GBytes   839 Mbits/sec                  
> receiver
> 
> iperf Done.
> 
> SENDER END


Before opening a ticket with a national provider, you may want to work on the 
local ends to do a couple of things:

 - Ensure that the local path(s) are clean.  Start at the host level, is it 
connected to a switch that has been properly tuned? Does it have a bottleneck 
free (e.g. if its a 10G host, is it free of any 1G stepdowns) path through 
the T2 and Campus network?  Are there any firewalls in the path that will 
impact TCP performance?

 - As your regional providers (IU GigaPoP/CENIC) if they have perfsonar 
testers available to help divide the path?  I know CENIC has many, unsure of 
what is available within the IU Gigapop.  

 - Test against the Internet2 hosts in Chicago/Kansas City/Houston/Los 
Angeles.  The goal of splitting up the path is to find the longest, cleanest, 
test you can.  When things start to degrade, you are centering in on where 
the problem may be.

Not to sound accusatory or point blame, but the common cause of problems of 
this nature is normally closer to the ends than the middle.  There are 
examples of where a backbone/regional device may be failing, but one would 
hope they are all using perfSONAR and caught it before it hurt the end user :)

Thanks;

-jason

On Jul 30, 2014, at 2:01 PM, Matthew J Zekauskas 
<>
 wrote:

> A note to Research Support 
> <>
>  should help for Internet2-related queries; if there was a particular 
> backbone issue found, someone from an end site (or regional) contacting the 
> Internet2 NOC would also be appropriate.
> 
> I don't believe latency is the issue here.    When using mtr, or 
> traceroute, be aware that the routers will preferentially do other tasks to 
> responding to ping or TTL=0 events, so the reported latency can be higher 
> (occasionally much higher) than transiting packets experience.
> 
> FWIW, from a site in los angeles to chicago (pc's near the router) the RTT 
> is 58.3 ms; from los angeles to kansas city it is 47.2 ms, which is in-line 
> with the measurements below.   chicago to kans is 11.1 to 11.2, which 
> matches the measurements from the other direction.  I might be able to dig 
> up some of the rough fiber mileage measurements that show this is basically 
> speed of light through fiber distance, but the numbers I have don't account 
> for a bunch of metro fiber.
> 
> I looked at the maddash owamp UI, and it looks like there are no packet 
> drops detected along the path, either.
> 
> --Matt.
> 
> On 7/30/14, 12:40 PM, Manoj Kumar Jha wrote:
>> Hello,
>>    We don't know whether this is an appropriate forum to raise this issue. 
>>  If not, please help us in diverting it to proper mailing list.
>> 
>> 
>> The perfsonar throughput from Purdue to Caltech is in lower range (~ 600 
>> Mbps) if we compare to other sites.   Throughput between Purdue and 
>> Caltech can be found on following link
>> 
>> http://tinyurl.com/oc95jzj
>> 
>> 
>> Following is the output from the tool 'mtr' from Purdue to Caltech 
>> perfsonar boxes.
>> 
>> 
>> """
>> [root@perfsonar-cms2
>>  ~]# mtr --report perfsonar.ultralight.org
>> HOST: perfsonar-cms2.itns.purdue. Loss%   Snt   Last   Avg  Best Wrst StDev
>>  1. math-g109-c9508-01-vlan743.t  0.0%    10    0.4   0.4   0.4 0.6   0.0
>>  2. math-g190-c7710-02-e2-4.tcom  0.0%    10    1.5   0.8   0.5 1.5   0.3
>>  3. math-g190-c7710-01-e2-1.tcom  0.0%    10    0.7   0.6   0.4 0.8   0.1
>>  4. tel-210-c9006-01-hu0-1-0-1.t  0.0%    10    0.7   0.7   0.6 0.9   0.1
>>  5. indiana-gigapop-ctc-internet  0.0%    10    1.6   1.6   1.6 1.8   0.1
>>  6. et-10-0-0.101.rtr.chic.net.i  0.0%    10    6.4   6.4   6.1 6.9   0.3
>>  7. et-10-0-0.106.rtr.kans.net.i  0.0%    10   17.6  17.6  17.3 18.1   0.3
>>  8. et-1-0-0.109.rtr.hous.net.in  0.0%    10   32.0  32.3  31.9 32.8   0.3
>>  9. hpr-lax-hpr2--i2-houston.cen  0.0%    10   65.2  67.3  64.5 78.0   5.3
>> 10. hpr-svl-ln-peer9.cenic.net    0.0%    10   65.9  65.6  65.3 65.9   0.2
>> 11. perfsonar.ultralight.org      0.0%    10   65.5  65.5  65.1 66.0   0.3
>> [root@perfsonar-cms2
>>  ~]#
>> 
>> """
>> 
>> 
>> 
>> Packet round trip time for Chiacgo (chic.net) -->  Kansas (kans.net) --> 
>> Houston (hous.net.in and houston.cent)  is higher if we compare to other 
>> hops. In the reverse direction, packet rtt is higher between 'lax-hpr' and 
>> 'kans.net'.  Following is the output of the tool 'traceroute'  from 
>> Caltech to Purdue perfsonar boxes.
>> ^^^
>> 
>> [root@gridftp1
>>  ~]# traceroute  perfsonar-cms2.itns.purdue.edu
>> traceroute to perfsonar-cms2.itns.purdue.edu (128.211.143.4), 30 hops
>> max, 60 byte packets
>> 1  ve64-chopin.86.84.192.in-addr.arpa (192.84.86.97)  0.209 ms 0.245
>> ms  0.277 ms
>> 2  hpr-svl-ln-peer8.cenic.net (137.164.26.106)  12.134 ms  1.222 ms  
>> 12.252 ms
>> 3  hpr-i2-newnet--lax-hpr.cenic.net (137.164.26.134)  1.279 ms 1.277
>> ms 137.164.26.201 (137.164.26.201)  1.248 ms
>> 4  et-1-0-0.111.rtr.hous.net.internet2.edu (198.71.45.20)  33.567 ms
>> 33.517 ms  33.498 ms
>> 5  et-5-0-0.109.rtr.kans.net.internet2.edu (198.71.45.17)  48.426 ms
>> 48.420 ms  48.401 ms
>> 6  et-9-0-0.106.rtr.chic.net.internet2.edu (198.71.45.14)  59.527 ms
>> 59.298 ms  59.256 ms
>> 7  ae-1.2063.rtr.ictc.indiana.gigapop.net (149.165.254.185) 63.895
>> ms  64.124 ms  64.126 ms
>> 8  * * *
>> 9  * * *
>> 10  * * *
>> 11  perfsonar-cms2.itns.purdue.edu (128.211.143.4)  64.952 ms 64.955
>> ms  65.289 ms
>> 
>> 
>> ^^^
>> Is the packet rtt value to/from Kansas hop is under normal value ?
>> 
>> Thanks,
>> Manoj