perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Greetings Joon & All;

Just for reference, ESnet has 3 testers at equinix peering locations at Ashburn, Chicago, and San Jose:

    eqx-ash-pt1.es.net
    eqx-chi-pt1.es.net
    eqx-sj-pt1.es.net

These follow the same rules as other ESnet testers (things on the R&E routing table can reach these), including several Amazon routes. 

The last time I did any Amazon testing (and as you note), we found that the LAN bandwidth (and by extension WAN bandwidth) was heavily gated by the 'size' of the VM you purchased.  Bigger is better, naturally.  Tuning doesn't help much, and it appeared that something in the path was selectively grooming TCP through loss.

Thanks;

-jason

Hyojoon Kim wrote:

Hello Matt, 

We are planning to get an AWS Direct Connect, from our campus to AoA in NYC. And we wanted to see how performance differs to Amazon services, before and after we light up that connection. 

So far, it seems we are bottlenecked by the AWS instance resource (bandwidth test to a t2.large instance shows around 615 Mbps). I was wondering if there was a public AWS instance with perfsonar installed and lots of resources (e.g., m10xl), which people can use to measure their network performance to AWS (not bottlenecked by other resources). 

If there is no standard or widely-used method (or a public perfsonar node in AWS), I guess the method you used about a year ago is the way most people do? (i.e., spin up an m10xl image and tune it)

Thank you for your reply. These are useful information! 

Thanks,
Joon

On Apr 2, 2017, at 12:02 PM, Matthew J Zekauskas  wrote:

Joon:

Can you say a bit more about what you are trying to understand?   Do you have a particular performance goal?  Going to a test point outside AWS will prove your infrastructure (which is definitely a good thing to do!), but it may say nothing about what you can practically get into (or out of ) AWS, since that depends on the AWS infrastructure (as you note).

Internet2 AL3S (R&E) has 80G in a 2 40G LAGs towards us-east-1 (Ashburn, also gives connectivity to the new Ohio region), and 20G toward us-west-2 (Seattle/Oregon).   The AL3S TR-CPS service has additional peering points.

pas-adhoc.seat.net.internet2.edu sits on the router that peers with us-west-2.   Internet2 itself doesn't have anything public currently in/around Ashburn, but there is a node we use internally at Ashburn that can likely be used for debugging if you contact the Internet2 NOC.  ESnet has a node in DC which is not too far from Ashburn, but I'm not sure how it is topologically to you.  (wash-pt1.es.net). MAX may have some nodes too.

I did some iperf testing to some AWS instances about a year ago, and maxed out around 4-5G toward Ashburn using an m10xl image.  You did also need to perform typical TCP tuning on the image.

One other thing to watch for is that all Amazon peerings use MTU 1500 (it's their restriction); however most R&E peerings are MTU 9000, and AWS uses MTU 9000 internally.   I have seen PMTUD blackholes causing really crappy performance in certain cases to R&E; so you may want to restrict the AWS test node to MTU 1500 for best performance.

--Matt (note, I'll be mostly sequestered Monday and Tuesday, so if you ask followup questions of me, I may not respond until after that...)

On 4/1/17 2:07 PM, Hyojoon Kim wrote:

Hello all,

Does anyone if there is a good perfSONAR node in or close by to an Amazon cloud service (AWS) spot (east, west, etc) so that we can measure how much throughput we get to AWS instances from our campus network?

We tried creating an instance ourselves and installing perfSONAR, which works. However, it seems to be bottlenecked by the AWS instance's resource, not network connection.  (e.g., http://stackoverflow.com/questions/18507405/ec2-instance-typess-exact-network-performance) 

Thanks,
Joon