perfSONAR User Q&A and Other Discussion

[Pol Llovet <>]

Ok,so we have a little bit more information, and i think that oar.net may be a part of the problem.  Note: I am using 1Gb speed links.

My traceroute to my initial IP (stinger.wvstateu.edu) and the closest perfsonar machine (perf.net.ohio-state.edu) share the first 11 hops (they diverge at clmbn-r5-et-8-0-0s101.core.oar.net).  The last perfsonar machine that I can test is hop 9 (wash.net.internet2.edu), see below for more info.

I get consistent 800Mb/600Mb out/in throughput to wash.net.internet2.edu, which is normal. The researcher at the initial IP (stinger.wvstateu.edu) states that he gets only 1MB/s downloads, but he doesn't have a perfsonar machine at his institution.

Tests to perf.net.ohio-state.edu show 700Mb/10Mb out/in throughput, which indicates to me that something is going on.  It looks like my outbound bandwidth to them is high, but the inbound is incredibly slow, which will affect TCP/IP transfers. 

Though, I am still not sure why I get a timeout in the traceroute to stinger.wvstateu.edu (hop 13 for me, see below).

Would someone else be willing to try a few bwctl tests to perf.net.ohio-state.edu and see if they also get slow inbound speeds?

Here are the specific commands I am running:

outbound test: bwctl -c perf.net.ohio-state.edu -f m

inbound test: bwctl -s perf.net.ohio-state.edu -f m

Here are the relevant traceroute segments:

stinger.wvstateu.edu:

 9  et-9-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57)  80.198 ms  80.604 ms  80.512 ms

10  clevs-r5-et-4-0-0s1109.core.oar.net (192.88.192.237)  88.976 ms  88.965 ms  89.089 ms

11  clmbn-r5-et-8-0-0s101.core.oar.net (192.153.39.253)  92.695 ms  92.609 ms  92.665 ms

12  clmbn-r9-xe-1-2-0s101.core.oar.net (192.153.38.22)  92.576 ms  92.083 ms  92.027 ms

13  * * *

14  129.71.255.165 (129.71.255.165)  95.818 ms  93.401 ms  93.824 ms

15  129.71.255.237 (129.71.255.237)  92.368 ms  92.623 ms  92.687 ms

16  stinger.wvstateu.edu (129.71.128.60)  92.611 ms !X  92.585 ms !X  93.201 ms !X

perf.net.ohio-state.edu:

 9  et-9-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57)  81.333 ms  82.942 ms  83.175 ms

10  clevs-r5-et-4-0-0s1109.core.oar.net (192.88.192.237)  88.634 ms  88.605 ms  88.575 ms

11  clmbn-r5-et-8-0-0s101.core.oar.net (192.153.39.253)  92.228 ms  92.528 ms  92.263 ms

12  clmbn-r0-xe-1-1-0s101.core.oar.net (192.153.38.14)  92.249 ms  92.343 ms  94.600 ms

13  clmbk-r9-xe-1-0-0s101.core.oar.net (192.153.38.26)  93.164 ms  92.687 ms  92.689 ms

14  clmbt-r9-xe-0-0-0s101.bb.oar.net (192.153.38.38)  92.684 ms  92.734 ms  92.770 ms

15  tc6-teng2-4s333.ohio-dmz.net (192.148.242.206)  92.612 ms  92.575 ms  92.618 ms

16  tc1-teng6-2.net.ohio-state.edu (164.107.2.253)  223.076 ms  219.513 ms  204.155 ms

17  kc1-teng2-10.net.ohio-state.edu (164.107.2.82)  92.351 ms  92.369 ms  95.831 ms

18  perf.net.ohio-state.edu (140.254.68.10)  92.497 ms  92.424 ms  92.370 ms

Thanks for any help you might have...

-pol

Pol M. Llovet
Research Computing / Middleware

Montana State University
w: 406-994-3416
c: 406-646-6747

On Tue, Aug 5, 2014 at 9:15 AM, Eli Dart <> wrote:

Hmmmm....

OARnet doesn't seem to have perfSONAR servers in the directory :(

I would ask on the list to see if anyone knows of any.

Also, I put "ohio" into http://stats.es.net/ServicesDirectory/ and came up with one box: perf.net.ohio-state.edu.

Looking at a traceroute, it appears that that host shares a portion of the OARnet path with the traffic you care about.

If you can get good performance to/from that host, you can mark the common portion of the path as "likely clean" (this would be hops 10 and 11 in your traceroute, if your path to the host is the same as mine).

Other than that, someone is going to need to deploy a test host somewhere.  Ideally, one would be at the handoff from OARnet to wvnet, and one at the handoff from wvnet to wvstateu.edu.

(also, I would keep this thread on the open list - more chance of others with local/regional knowledge being able to help).

Eli

On Mon, Aug 4, 2014 at 11:50 PM, Pol Llovet <> wrote:

Sure, here's my traceroute:

perfsonar:~ $ traceroute 129.71.128.60

traceroute to 129.71.128.60 (129.71.128.60), 30 hops max, 60 byte packets

 1  n7k-ajmps1.msu.montana.edu (153.90.1.14)  0.494 ms  0.581 ms  0.856 ms

 2  192.105.205.192 (192.105.205.192)  1.222 ms  1.100 ms  1.180 ms

 3  xe-1-0-0--566.icar-spknwa01-01.infra.pnw-gigapop.net (209.124.181.246)  8.072 ms  8.073 ms  8.057 ms

 4  64.57.28.54 (64.57.28.54)  15.476 ms  15.537 ms  15.587 ms

 5  xe-0-3-0.10.rtr.seat.net.internet2.edu (64.57.28.53)  15.505 ms  15.497 ms  15.486 ms

 6  et-4-0-0.113.rtr.salt.net.internet2.edu (198.71.45.24)  31.443 ms  31.431 ms  31.550 ms

 7  et-5-0-0.110.rtr.kans.net.internet2.edu (198.71.45.18)  51.677 ms  51.836 ms  51.841 ms

 8  et-9-0-0.106.rtr.chic.net.internet2.edu (198.71.45.14)  62.918 ms  62.939 ms  62.897 ms

 9  et-9-0-0.115.rtr.wash.net.internet2.edu (198.71.45.57)  79.999 ms  79.655 ms  79.761 ms

10  clevs-r5-et-4-0-0s1109.core.oar.net (192.88.192.237)  88.865 ms  88.873 ms  88.862 ms

11  clmbn-r5-et-8-0-0s101.core.oar.net (192.153.39.253)  92.698 ms  92.502 ms  92.507 ms

12  clmbn-r9-xe-1-2-0s101.core.oar.net (192.153.38.22)  92.492 ms  92.460 ms  92.433 ms

13  * * *

14  129.71.255.165 (129.71.255.165)  92.123 ms  92.074 ms  92.177 ms

15  129.71.255.237 (129.71.255.237)  93.185 ms  92.100 ms  92.261 ms

16  stinger.wvstateu.edu (129.71.128.60)  92.632 ms !X  92.784 ms !X  92.710 ms !X

And like I said, I get great test results all the way to wash.net.internet2.edu, but I am not sure how to test from hop 9 to 16.

Pol M. Llovet
Research Computing / Middleware

Montana State University
w: 406-994-3416
c: 406-646-6747

On Mon, Aug 4, 2014 at 4:32 PM, Eli Dart <> wrote:

What location in West Virginia?

Do you have a traceroute?

Eli

On Mon, Aug 4, 2014 at 3:24 PM, Pol Llovet <> wrote:

Hi, we are diagnosing a slow throughput situation to West Virginia. All the tests through to the Washington DC internet2 PS node look good (bwctl.wash.net.internet2.edu).  I would like to test from that node through to the endpoint.  

The endpoint is just a workstation IP, and there aren't any perfsonar nodes on-campus that I can test against.  I was thinking I could just do a traceroute/ping test from the node UI, but it doesn't have the UI exposed for testing (that I could find).  And since I don't have a PS endpoint at one end, I can't use the bw* cli tools.

Any tips on how I could diagnose the rest of this issue?

Pol M. Llovet
Research Computing / Middleware

Montana State University
w: 406-994-3416
c: 406-646-6747

--

Eli Dart, Network Engineer                          NOC: (510) 486-7600

ESnet Office of the CTO (AS293)                          (800) 333-7638

Lawrence Berkeley National Laboratory 

PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3

--

Eli Dart, Network Engineer                          NOC: (510) 486-7600

ESnet Office of the CTO (AS293)                          (800) 333-7638

Lawrence Berkeley National Laboratory 

PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3