perfSONAR User Q&A and Other Discussion

[Brian Tierney <>]

On Wednesday, January 27, 2016, Rittenhouse, Robert (LACA) <> wrote:

I am having issues manually running owping between the server at our central location and the test endpoint that I have set up in the field. That (probably firewall) issue aside for now, iperf3 did report when I ran 20 parallel instances that it was able to reach 288Mbps combined (the circuit is 500 Mbps). There were almost 4,000 TCP re-transmits though. I'm searching around to see how many re-transmits are reasonable but I haven't yet come upon that information.

basically if you want good performance you need zero loss/retransmits. 

See: https://fasterdata.es.net/network-tuning/tcp-issues-explained/packet-loss/

 

Have you looked at utilization data? How congested is that path?

We have a Cisco 6500 core with a Firewall Services Module. The perfsonar toolkit server is installed on a server that is connected to a port on the 6500. The data then has to traverse through the FWSM module in the 6500 and over to our customers network where I have this device. We have a VPLS (I believe) circuit that's basically a layer 2 point-to-point fiber circuit that spits out at the customers location into a Cisco 3560. The testpoint node hangs off of a port on our 3560 CPE gear.

[Customer LAN] <===>[Cisco 3560] <===> (Time Warner Cloud) <===> Cisco 6500 <=====> [perfsonar server]

                                            ^[testpoint]

There is data on the circuit almost 24x7 but it goes down during the late late night/mid morning after backups are done running on this particular circuit. For instance, today there was a maximum of (30 second average) 150Mbps (total) passing through it at one point.

Does that help at all? The perfsonar toolkit web ui acts weird too. It seems like if I create a TCP throughput test and then I go back and change it to UDP, it doesn't seem to want to work. I have to re-create the test it seems. I just created a second throughput test (to the same testpoint host). Shouldn't there be a second test listed under test results? Or is it only 1 because it's the same destination? It acts very strange sometimes but that's a different email.

Thanks,

Robert Rittenhouse
Network Coordinator
Licking Area Computer Association (LACA)
(740) 345-3400 ext. 220

---

From: Brian Tierney []
Sent: Wednesday, January 27, 2016 10:05 AM
To: Rittenhouse, Robert (LACA)
Cc: Szymon Trocha;
Subject: Re: [perfsonar-user] Forcing parallel flows for Throughput test

Some other things to check:

does owping show packet loss?

does iperf3 should TCP retransmits?

If so, then you likely have congestion or bad connections on that path.

see: https://fasterdata.es.net/performance-testing/troubleshooting/network-troubleshooting-quick-reference-guide/

You might also want to capture a tcpdump to look at what TCP is doing in detail:

https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/tcpdump-tcptrace/

On Wed, Jan 27, 2016 at 5:53 AM, Rittenhouse, Robert (LACA) <> wrote:

I will look into the host tuning more. I just didn't want to lock myself out of the testpoint gear and have to take a nice, long, relaxing drive away from the office to fix it ;) 

I was able to run a bwctl command that when I told it to use iperf3 and to run 20 parallel processes and that got us up to 288Mbps. I just know with some of our time warner links, when we test, we seem to be limited to how big each flow can be. When we test back to our same iperf gear here (and the other way around) on some of our Time Warner fiber circuits, they always let a single stream go to it's full potential. I will surely try these host tuning settings and get back with you. It's rather frustrating when link testing...

Thank you,

Robert Rittenhouse
Network Coordinator
Licking Area Computer Association (LACA)
(740) 345-3400 ext. 220

---

From: [] on behalf of Szymon Trocha []
Sent: Wednesday, January 27, 2016 3:47 AM
To: Rittenhouse, Robert (LACA)
Cc:
Subject: Re: [perfsonar-user] Forcing parallel flows for Throughput test

W dniu 26.01.2016 o 19:01, Rittenhouse, Robert (LACA) pisze:

I have a Perfsonar Centos Toolkit server set up as a central manager for (hopefully) automating our throughput and latency testing that we only do reactively.

My issue is, when I add a standard TCP throughput test to a remote Ubuntu perfsonar-testpoint machine, it's only able to pull about 25Mbps max for a single stream. There is a point-to-point fiber link between my building where the Central test server is and where the testpoint resides. We had this issue with iperf and we had to have it use a lot of processes to try and max the link out. It is a 500 Mbps link if that helps.

How would I accomplish this task with perfsonar and keep it automated?

Hi Robert,

There are many factors which may limit your throughput. Are there any network segments and network devices between the two systems? Is there any concurrent traffic on the links between the two systems?
Did you look at http://docs.perfsonar.net/manage_tuning.html for host tuning?

Regards,

-- 
Szymon Trocha

Poznań Supercomputing & Netw. Center ::: NETWORK OPERATION CENTER
Tel. +48 618582022 ::: http://noc.man.poznan.pl

---

PLEASE NOTE: This message and any response to it may constitute a public record, and therefore may be available upon request in accordance with Ohio public records law. (ORC 149.43)

This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain private, confidential, and/or privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, employee, or agent responsible for delivering this message, please contact the sender by reply e-mail and destroy all copies of the original e-mail message.

--

Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net

---

PLEASE NOTE: This message and any response to it may constitute a public record, and therefore may be available upon request in accordance with Ohio public records law. (ORC 149.43)

This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain private, confidential, and/or privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, employee, or agent responsible for delivering this message, please contact the sender by reply e-mail and destroy all copies of the original e-mail message.

-- 
Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net