perfSONAR User Q&A and Other Discussion

[Matthew J Zekauskas <>]

If you want me to help facilitate tests to Internet2's ASHB test point, let me know.   You are correct, it's not public, and is currently connected to a Layer2-only device.

The traceroutes you gave did show the same path to both addresses within Amazon.

I have some experience getting iperf & bwctl to work inside an AWS VM, if that becomes useful.  I think I can dig up some use of owamp within AWS as well, although I have no personal experience.

While not perfsonar per-se, you can set up a small VM within AWS to respond to ping, and given the long periods of loss you report, just setting up a long-running ping with say a 15 second interval may help with independent end-to-end validation.

Finally, while somewhat long in the tooth, there is pas-adhoc.chic.net.internet2.edu if you want something adjacent to rtsw.chic.net.internet2.edu as a test point.

--Matt

On 1/31/17 1:40 PM, Casey Russell wrote:

Thanks Jason,

     I'd located your node also, I just don't know enough about the actual infrastructure out there to know how similar the paths were once the traffic left Chicago.  As it is, it looks like my two best options are you guys or the I2 node, and since you guys will allow the tests today and I2 won't (without me contacting someone and trying to organize it), you win.  :-)

     Thank you, I'll set up some testing and I'll at least get the visibility on the path I need up to the point where it leaves Chicago.

I appreciate the help

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Mon, Jan 30, 2017 at 10:58 AM, Jason Zurawski <> wrote:

Greetings Casey;

There is an ESnet tester that is located on the ESnet side of the demarcation of our peering to Equinix in Ashburn: eqx-ash-pt1.es.net.  Unclear if this will be exactly what you need, but here is the routing info between it and your Bryant tester:

[rootjz@atla-pt1 ~]# bwtraceroute -c ps-bryant-bw.perfsonar.kanren.net -s eqx-ash-pt1.es.net
bwtraceroute: Using tool: traceroute
bwtraceroute: 26 seconds until test results available

SENDER START
traceroute to 164.113.32.141 (164.113.32.141), 30 hops max, 60 byte packets
 1  eqxashrt1-eqxashpt1.es.net (198.124.238.81)  0.166 ms  0.174 ms  0.172 ms
 2  washcr5-ip-a-eqxashrt1.es.net (134.55.221.74)  1.212 ms  1.599 ms washcr5-ip-b-eqxashrt1.es.net (134.55.221.70)  1.937 ms
 3  198.124.194.10 (198.124.194.10)  0.925 ms  0.920 ms  0.911 ms
 4  et-7-1-0.4070.rtsw.chic.net.internet2.edu (198.71.45.56)  18.072 ms  18.079 ms  18.066 ms
 5  et-5-0-0.4070.rtsw.kans.net.internet2.edu (198.71.45.15)  28.484 ms  28.490 ms  28.480 ms
 6  64.57.28.178 (64.57.28.178)  28.462 ms  28.472 ms  28.489 ms
 7  bb-kc-grand-et0-0-0-0.bb.net.kanren.net (164.113.192.41)  28.536 ms  28.548 ms  28.545 ms
 8  ps-bryant-bw.perfsonar.kanren.net (164.113.32.141)  28.449 ms  28.452 ms  28.447 ms

SENDER END
[rootjz@atla-pt1 ~]# bwtraceroute -s ps-bryant-bw.perfsonar.kanren.net -c eqx-ash-pt1.es.net
bwtraceroute: Using tool: traceroute
bwtraceroute: 26 seconds until test results available

SENDER START
traceroute to 198.124.238.82 (198.124.238.82), 30 hops max, 60 byte packets
 1  164.113.32.142 (164.113.32.142)  0.214 ms  0.215 ms  0.205 ms
 2  bb-kc-walnut-et-11-0-0-0.bb.net.kanren.net (164.113.192.42)  0.208 ms  0.225 ms  0.217 ms
 3  et-10-0-0.1050.rtsw.kans.net.internet2.edu (64.57.28.177)  0.296 ms  0.313 ms  0.308 ms
 4  et-11-1-0.4070.rtsw.chic.net.internet2.edu (198.71.45.14)  11.235 ms  11.235 ms  11.230 ms
 5  198.71.45.155 (198.71.45.155)  11.913 ms  12.187 ms  12.553 ms
 6  starcr5-ip-a-chiccr5.es.net (134.55.42.42)  11.695 ms washcr5-ip-a-chiccr5.es.net (134.55.36.46)  28.745 ms starcr5-ip-a-chiccr5.es.net (134.55.42.42)  11.700 ms
 7  eqxashrt1-ip-a-starcr5.es.net (134.55.36.102)  28.368 ms eqxashrt1-ip-b-starcr5.es.net (134.55.37.66)  55.007 ms  55.012 ms
 8  eqx-ash-pt1.es.net (198.124.238.82)  28.398 ms  28.373 ms  28.368 ms

SENDER END

Note that this (and the other) ESnet testers are only open to the R&E routing table and a select few ranges from the commercial space, e.g. if you are testing to something and it fails, that will be the reason. 

Thanks;

-jason

Casey Russell wrote:

Group,

     We're helping a KanREN member institution to troubleshoot some connectivity problems with one of their campus applications, and, as is often the case, it has become necessary to try to prove fault with as much precision as possible.  The application is in the cloud and is hosted in Amazon's Ashburn, VA data centers.  I searched around a bit on Friday looking for anyone hosting PS nodes in Ashburn, near the AWS datacenters and actually found one run by Internet2, and I could hit them up about testing to it, however, it uses a slightly different (AL2S) path through Cleveland, than our member's traffic does.  

     So I wondered if anyone else in the community knows of PS nodes sitting on the I2 network in the Ashburn, VA area, specifically hanging off of (or otherwise downstream from) the I2 router there.  I'm looking for a device that's in the standard I2 RnE table so that it will follow the same path as my member traffic.  I'd like to set up some temporary latency/loss testing for a week or two to see if my packet loss testing matches their (the member's) monitoring system.  I realize not everyone publishes their nodes in the global directory or makes them available for public testing, so if you'd prefer to reply to me off-list, that's fine, I'd appreciate the assistance.

     I'd also appreciate other thoughts and ideas, (i.e. anyone who's had luck deploying a temporary PS node on an AWS vm, or other "outside the box" thinking that I might not have considered).

I included a representative traceroute from within our network (although not from within our member network) in case someone can offer a node that's a step or two back from Ashburn (chicago area) and might still be of assistance. 

Thanks everyone. 

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

Problem description:  During multiple periods during the workday, they experience 1-8 minute periods where packet loss goes from effectively zero, to 90+ percent.  It will stay there for the 1-8 minutes and then the problem corrects its self entirely.  

Hosted IP #1

crussell@bb-kc-walnut> traceroute 52.21.4.124   

traceroute to 52.21.4.124 (52.21.4.124), 30 hops max, 48 byte packets

 1  et-10-0-0.1050.rtsw.kans.net.internet2.edu (64.57.28.177)  0.588 ms  0.542 ms  0.550 ms

 2  et-11-1-0.4070.rtsw.chic.net.internet2.edu (198.71.45.14)  11.456 ms  11.725 ms  11.465 ms

 3  et-1-0-0.120.rtr.ashb.net.internet2.edu (64.57.28.10)  28.010 ms  28.039 ms  28.025 ms

 4  64.57.30.39 (64.57.30.39)  28.262 ms 64.57.30.3 (64.57.30.3)  28.576 ms  28.092 ms

 5  * * *

 6  * * *

 7  54.239.110.157 (54.239.110.157)  47.009 ms 54.239.110.185 (54.239.110.185)  51.603 ms 54.239.110.149 (54.239.110.149)  49.070 ms

*****Quitting here, because we've entered AWS domain*****

Hosted IP#2 

crussell@bb-kc-walnut> traceroute 52.4.40.57                        

traceroute to 52.4.40.57 (52.4.40.57), 30 hops max, 48 byte packets

 1  et-10-0-0.1050.rtsw.kans.net.internet2.edu (64.57.28.177)  0.564 ms  0.550 ms  0.562 ms

 2  et-11-1-0.4070.rtsw.chic.net.internet2.edu (198.71.45.14)  11.454 ms  11.468 ms  11.510 ms

 3  ge-1-0-0.518.rtr.chic.net.internet2.edu (64.57.28.10)  28.029 ms  28.020 ms  28.024 ms

 4  64.57.30.39 (64.57.30.39)  28.105 ms  31.207 ms  28.588 ms

 5  * * *

 6  * * *

 7  54.239.110.191 (54.239.110.191)  31.768 ms 54.239.110.163 (54.239.110.163)  35.461 ms 54.239.110.187 (54.239.110.187)  32.206 ms

*****Quitting here, because we've entered AWS domain*****