perfSONAR User Q&A and Other Discussion

["Rittenhouse, Robert (LACA)" <>]

Yeah 'sysopt np completion-unit' is configured.

I should see if we have a box that I can install the perfsonar-testpoint that 
is behind a good working circuit and compare the results. I know some of our 
local Time Warner circuits will test correct with just a single iperf 
instance. I assume those are the non-congested links.

Thank you for the suggestion.

Brian Tierney - As for utilization, the link would only be utilized at the 
maximum of 200 Mbps on a normal day and it's a 500 Mbps circuit.

Robert Rittenhouse
Network Coordinator
Licking Area Computer Association (LACA)
(740) 345-3400 ext. 220


________________________________________
From: Dale W. Carder 
[]
Sent: Wednesday, January 27, 2016 5:41 PM
To: Rittenhouse, Robert (LACA)
Cc: 
;
 Szymon Trocha; 

Subject: Re: [perfsonar-user] Forcing parallel flows for Throughput test

Just a quick sanity check as I see the words FWSM and retransmits below.
On the FWSM, do you have 'sysopt np completion-unit' configured?

best,
Dale

Thus spake Rittenhouse, Robert (LACA) 
()
 on Wed, Jan 27, 2016 at 09:31:18PM +0000:
> I am having issues manually running owping between the server at our 
> central location and the test endpoint that I have set up in the field. 
> That (probably firewall) issue aside for now, iperf3 did report when I ran 
> 20 parallel instances that it was able to reach 288Mbps combined (the 
> circuit is 500 Mbps). There were almost 4,000 TCP re-transmits though. I'm 
> searching around to see how many re-transmits are reasonable but I haven't 
> yet come upon that information.
>
> We have a Cisco 6500 core with a Firewall Services Module. The perfsonar 
> toolkit server is installed on a server that is connected to a port on the 
> 6500. The data then has to traverse through the FWSM module in the 6500 and 
> over to our customers network where I have this device. We have a VPLS (I 
> believe) circuit that's basically a layer 2 point-to-point fiber circuit 
> that spits out at the customers location into a Cisco 3560. The testpoint 
> node hangs off of a port on our 3560 CPE gear.
>
> [Customer LAN] <===>[Cisco 3560] <===> (Time Warner Cloud) <===> Cisco 6500 
> <=====> [perfsonar server]
>                                             ^[testpoint]
>
> There is data on the circuit almost 24x7 but it goes down during the late 
> late night/mid morning after backups are done running on this particular 
> circuit. For instance, today there was a maximum of (30 second average) 
> 150Mbps (total) passing through it at one point.
>
> Does that help at all? The perfsonar toolkit web ui acts weird too. It 
> seems like if I create a TCP throughput test and then I go back and change 
> it to UDP, it doesn't seem to want to work. I have to re-create the test it 
> seems. I just created a second throughput test (to the same testpoint 
> host). Shouldn't there be a second test listed under test results? Or is it 
> only 1 because it's the same destination? It acts very strange sometimes 
> but that's a different email.
>
> Thanks,
>
> Robert Rittenhouse
> Network Coordinator
> Licking Area Computer Association (LACA)
> (740) 345-3400 ext. 220
> <mailto:>
> ________________________________
> From: Brian Tierney 
> []
> Sent: Wednesday, January 27, 2016 10:05 AM
> To: Rittenhouse, Robert (LACA)
> Cc: Szymon Trocha; 
> 
> Subject: Re: [perfsonar-user] Forcing parallel flows for Throughput test
>
>
> Some other things to check:
>
> does owping show packet loss?
> does iperf3 should TCP retransmits?
>
> If so, then you likely have congestion or bad connections on that path.
>
> see: 
> https://fasterdata.es.net/performance-testing/troubleshooting/network-troubleshooting-quick-reference-guide/
>
> You might also want to capture a tcpdump to look at what TCP is doing in 
> detail:
> https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/tcpdump-tcptrace/
>
>
>
> On Wed, Jan 27, 2016 at 5:53 AM, Rittenhouse, Robert (LACA) 
> <<mailto:>>
>  wrote:
> I will look into the host tuning more. I just didn't want to lock myself 
> out of the testpoint gear and have to take a nice, long, relaxing drive 
> away from the office to fix it ;)
>
> I was able to run a bwctl command that when I told it to use iperf3 and to 
> run 20 parallel processes and that got us up to 288Mbps. I just know with 
> some of our time warner links, when we test, we seem to be limited to how 
> big each flow can be. When we test back to our same iperf gear here (and 
> the other way around) on some of our Time Warner fiber circuits, they 
> always let a single stream go to it's full potential. I will surely try 
> these host tuning settings and get back with you. It's rather frustrating 
> when link testing...
>
> Thank you,
> Robert Rittenhouse
> Network Coordinator
> Licking Area Computer Association (LACA)
> (740) 345-3400 ext. 220<tel:%28740%29%20345-3400%20ext.%20220>
> <mailto:>
> ________________________________
> From: 
> <mailto:>
>  
> [<mailto:>]
>  on behalf of Szymon Trocha 
> [<mailto:>]
> Sent: Wednesday, January 27, 2016 3:47 AM
> To: Rittenhouse, Robert (LACA)
> Cc: 
> <mailto:>
> Subject: Re: [perfsonar-user] Forcing parallel flows for Throughput test
>
> W dniu 26.01.2016 o 19:01, Rittenhouse, Robert (LACA) pisze:
> I have a Perfsonar Centos Toolkit server set up as a central manager for 
> (hopefully) automating our throughput and latency testing that we only do 
> reactively.
>
> My issue is, when I add a standard TCP throughput test to a remote Ubuntu 
> perfsonar-testpoint machine, it's only able to pull about 25Mbps max for a 
> single stream. There is a point-to-point fiber link between my building 
> where the Central test server is and where the testpoint resides. We had 
> this issue with iperf and we had to have it use a lot of processes to try 
> and max the link out. It is a 500 Mbps link if that helps.
>
> How would I accomplish this task with perfsonar and keep it automated?
>
>
>
> Hi Robert,
>
> There are many factors which may limit your throughput. Are there any 
> network segments and network devices between the two systems? Is there any 
> concurrent traffic on the links between the two systems?
> Did you look at http://docs.perfsonar.net/manage_tuning.html for host 
> tuning?
>
> Regards,
>
> --
> Szymon Trocha
>
> Poznań Supercomputing & Netw. Center ::: NETWORK OPERATION CENTER
> Tel. +48 618582022<tel:%2B48%20618582022> ::: http://noc.man.poznan.pl
>
> ________________________________
> PLEASE NOTE: This message and any response to it may constitute a public 
> record, and therefore may be available upon request in accordance with Ohio 
> public records law. (ORC 149.43)
>
> This e-mail message, including any attachments, is for the sole use of the 
> intended recipient(s) and may contain private, confidential, and/or 
> privileged information. Any unauthorized review, use, disclosure, or 
> distribution is prohibited. If you are not the intended recipient, 
> employee, or agent responsible for delivering this message, please contact 
> the sender by reply e-mail and destroy all copies of the original e-mail 
> message.
>
>
>
> --
> Brian Tierney, http://www.es.net/tierney
> Energy Sciences Network (ESnet), Berkeley National Lab
> http://fasterdata.es.net
>
> ________________________________
> PLEASE NOTE: This message and any response to it may constitute a public 
> record, and therefore may be available upon request in accordance with Ohio 
> public records law. (ORC 149.43)
>
> This e-mail message, including any attachments, is for the sole use of the 
> intended recipient(s) and may contain private, confidential, and/or 
> privileged information. Any unauthorized review, use, disclosure, or 
> distribution is prohibited. If you are not the intended recipient, 
> employee, or agent responsible for delivering this message, please contact 
> the sender by reply e-mail and destroy all copies of the original e-mail 
> message.
________________________________
 PLEASE NOTE: This message and any response to it may constitute a public 
record, and therefore may be available upon request in accordance with Ohio 
public records law. (ORC 149.43)

This e-mail message, including any attachments, is for the sole use of the 
intended recipient(s) and may contain private, confidential, and/or 
privileged information. Any unauthorized review, use, disclosure, or 
distribution is prohibited. If you are not the intended recipient, employee, 
or agent responsible for delivering this message, please contact the sender 
by reply e-mail and destroy all copies of the original e-mail message.