perfSONAR User Q&A and Other Discussion

["Dale W. Carder" <>]

Just a quick sanity check as I see the words FWSM and retransmits below.
On the FWSM, do you have 'sysopt np completion-unit' configured?

best,
Dale

Thus spake Rittenhouse, Robert (LACA) 
()
 on Wed, Jan 27, 2016 at 09:31:18PM +0000:
> I am having issues manually running owping between the server at our 
> central location and the test endpoint that I have set up in the field. 
> That (probably firewall) issue aside for now, iperf3 did report when I ran 
> 20 parallel instances that it was able to reach 288Mbps combined (the 
> circuit is 500 Mbps). There were almost 4,000 TCP re-transmits though. I'm 
> searching around to see how many re-transmits are reasonable but I haven't 
> yet come upon that information.
> 
> We have a Cisco 6500 core with a Firewall Services Module. The perfsonar 
> toolkit server is installed on a server that is connected to a port on the 
> 6500. The data then has to traverse through the FWSM module in the 6500 and 
> over to our customers network where I have this device. We have a VPLS (I 
> believe) circuit that's basically a layer 2 point-to-point fiber circuit 
> that spits out at the customers location into a Cisco 3560. The testpoint 
> node hangs off of a port on our 3560 CPE gear.
> 
> [Customer LAN] <===>[Cisco 3560] <===> (Time Warner Cloud) <===> Cisco 6500 
> <=====> [perfsonar server]
>                                             ^[testpoint]
> 
> There is data on the circuit almost 24x7 but it goes down during the late 
> late night/mid morning after backups are done running on this particular 
> circuit. For instance, today there was a maximum of (30 second average) 
> 150Mbps (total) passing through it at one point.
> 
> Does that help at all? The perfsonar toolkit web ui acts weird too. It 
> seems like if I create a TCP throughput test and then I go back and change 
> it to UDP, it doesn't seem to want to work. I have to re-create the test it 
> seems. I just created a second throughput test (to the same testpoint 
> host). Shouldn't there be a second test listed under test results? Or is it 
> only 1 because it's the same destination? It acts very strange sometimes 
> but that's a different email.
> 
> Thanks,
> 
> Robert Rittenhouse
> Network Coordinator
> Licking Area Computer Association (LACA)
> (740) 345-3400 ext. 220
> <mailto:>
> ________________________________
> From: Brian Tierney 
> []
> Sent: Wednesday, January 27, 2016 10:05 AM
> To: Rittenhouse, Robert (LACA)
> Cc: Szymon Trocha; 
> 
> Subject: Re: [perfsonar-user] Forcing parallel flows for Throughput test
> 
> 
> Some other things to check:
> 
> does owping show packet loss?
> does iperf3 should TCP retransmits?
> 
> If so, then you likely have congestion or bad connections on that path.
> 
> see: 
> https://fasterdata.es.net/performance-testing/troubleshooting/network-troubleshooting-quick-reference-guide/
> 
> You might also want to capture a tcpdump to look at what TCP is doing in 
> detail:
> https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/tcpdump-tcptrace/
> 
> 
> 
> On Wed, Jan 27, 2016 at 5:53 AM, Rittenhouse, Robert (LACA) 
> <<mailto:>>
>  wrote:
> I will look into the host tuning more. I just didn't want to lock myself 
> out of the testpoint gear and have to take a nice, long, relaxing drive 
> away from the office to fix it ;)
> 
> I was able to run a bwctl command that when I told it to use iperf3 and to 
> run 20 parallel processes and that got us up to 288Mbps. I just know with 
> some of our time warner links, when we test, we seem to be limited to how 
> big each flow can be. When we test back to our same iperf gear here (and 
> the other way around) on some of our Time Warner fiber circuits, they 
> always let a single stream go to it's full potential. I will surely try 
> these host tuning settings and get back with you. It's rather frustrating 
> when link testing...
> 
> Thank you,
> Robert Rittenhouse
> Network Coordinator
> Licking Area Computer Association (LACA)
> (740) 345-3400 ext. 220<tel:%28740%29%20345-3400%20ext.%20220>
> <mailto:>
> ________________________________
> From: 
> <mailto:>
>  
> [<mailto:>]
>  on behalf of Szymon Trocha 
> [<mailto:>]
> Sent: Wednesday, January 27, 2016 3:47 AM
> To: Rittenhouse, Robert (LACA)
> Cc: 
> <mailto:>
> Subject: Re: [perfsonar-user] Forcing parallel flows for Throughput test
> 
> W dniu 26.01.2016 o 19:01, Rittenhouse, Robert (LACA) pisze:
> I have a Perfsonar Centos Toolkit server set up as a central manager for 
> (hopefully) automating our throughput and latency testing that we only do 
> reactively.
> 
> My issue is, when I add a standard TCP throughput test to a remote Ubuntu 
> perfsonar-testpoint machine, it's only able to pull about 25Mbps max for a 
> single stream. There is a point-to-point fiber link between my building 
> where the Central test server is and where the testpoint resides. We had 
> this issue with iperf and we had to have it use a lot of processes to try 
> and max the link out. It is a 500 Mbps link if that helps.
> 
> How would I accomplish this task with perfsonar and keep it automated?
> 
> 
> 
> Hi Robert,
> 
> There are many factors which may limit your throughput. Are there any 
> network segments and network devices between the two systems? Is there any 
> concurrent traffic on the links between the two systems?
> Did you look at http://docs.perfsonar.net/manage_tuning.html for host 
> tuning?
> 
> Regards,
> 
> --
> Szymon Trocha
> 
> Poznań Supercomputing & Netw. Center ::: NETWORK OPERATION CENTER
> Tel. +48 618582022<tel:%2B48%20618582022> ::: http://noc.man.poznan.pl
> 
> ________________________________
> PLEASE NOTE: This message and any response to it may constitute a public 
> record, and therefore may be available upon request in accordance with Ohio 
> public records law. (ORC 149.43)
> 
> This e-mail message, including any attachments, is for the sole use of the 
> intended recipient(s) and may contain private, confidential, and/or 
> privileged information. Any unauthorized review, use, disclosure, or 
> distribution is prohibited. If you are not the intended recipient, 
> employee, or agent responsible for delivering this message, please contact 
> the sender by reply e-mail and destroy all copies of the original e-mail 
> message.
> 
> 
> 
> --
> Brian Tierney, http://www.es.net/tierney
> Energy Sciences Network (ESnet), Berkeley National Lab
> http://fasterdata.es.net
> 
> ________________________________
> PLEASE NOTE: This message and any response to it may constitute a public 
> record, and therefore may be available upon request in accordance with Ohio 
> public records law. (ORC 149.43)
> 
> This e-mail message, including any attachments, is for the sole use of the 
> intended recipient(s) and may contain private, confidential, and/or 
> privileged information. Any unauthorized review, use, disclosure, or 
> distribution is prohibited. If you are not the intended recipient, 
> employee, or agent responsible for delivering this message, please contact 
> the sender by reply e-mail and destroy all copies of the original e-mail 
> message.