perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi,

If you haven’t already, I’d recommend setting up some OWAMP tests and letting those run for awhile as those should give you a better picture of the loss events. If you are really worried about loss, setup some UDP throughput tests and see if they report any loss. TCP retransmits are an imperfect measure for loss so it’s hard to say. Some retransmits are perfectly normal, and can be the signal to TCP that it needs to back off. We don't actually display the breakdown of the test into one second intervals unfortunately, but its pretty common to see a handful of retransmits in the first few seconds of a transfer. You could also try setting the “omit” flag in your test to something like 5 seconds, and it will ignore any retransmits in the first 5 seconds of the test and help rule out some of the causes.

On ESnet we have mostly CentOS 6 boxes and a few CentOS 7 boxes (in the very near future that will be almost all CentOS 7). Its not un-common to see retransmits on both, but I don't have enough data points to say if there is something about the OS that’s more likely to cause them on 7. It also would not shock me if something about CentOS 7 is better able to report retransmits to iperf as well, so it could be a lot of factors. 

Thanks,

Andy

On August 23, 2017 at 9:24:55 AM, Thomas Tam () wrote:

Hi All,

 

Thanks for those who has responded to the Tao’s post.

 

I just want to give a context of why we are concerning the retransmission on Centos 7. We are looking into replacing all of our old perfSONAR servers, planning to buy around 30 of new servers. if these retransmissions are related to an inadequacy of the server hardware, we want to know before buying them. If this issue has been observed from a number of perfSONAR administrators after upgrading to Centos 7, perhaps this has nothing to do with the hardware specification. We would be comfortable to go ahead with the purchase without knowing the causes.

 

We have spent considerable amount of time to try various configurations. The latest test was done between two boxes connected back-to-back. This should eliminate the issues of packet loss, out-of-order and lack of memory buffer, but we were still seeing retransmissions. Honestly, we aren’t experts in perfSONAR and Centos, we relate on the expertise in the community helping us to make a right choice.

 

Your feedbacks are grealy appericated.

 

Best regards,

-Thomas

 

 

From: Tao Zhang <>
Date: Wednesday, August 23, 2017 at 8:52 AM
To: "Garnizov, Ivan (RRZE)" <>, Michael Johnson <>
Cc: "" <>
Subject: Re: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi All,

I understand pS is a measurement framework, not a measurement tool.  We have tried many test scenarios (different types of servers, different system parameter settings, different 10G NIC cards) trying to figure out why perfSONAR (iperf3) throughput tests have these TCP retransmissions on Centos 7.

Does anyone who is running perfSONAR on Centos 7 and not having these TCP retransmissions?

 

Thanks

 

Tao

 

 

From: "Garnizov, Ivan (RRZE)" <>
Date: Monday, August 7, 2017 at 6:49 AM
To: Tao Zhang <>, Michael Johnson <>
Cc: "" <>
Subject: RE: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi Tao,

 

IMHO there is a bit of a misunderstanding here. perfSONAR toolkit is not a measurement tool. pS is a measurement framework, which can coordinate measurements of multiple measurement tools and much more.

 

perfSONAR toolkit indeed provides some kernel and interface optimizations out of the box, but these are still generic and there is no such one-size-fits-all solution. After all you are reaching 10G.

 

You will have to go in the details and reach for the respective tool maintainers (depending on the tool you use) to get a clear understanding on the discrepancy you observe.

The difference between CentOS 6.9 and CentOS 7 is not only about the kernel. It is also about the drivers of the NIC.

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

 

 

From: [mailto:] On Behalf Of Tao Zhang
Sent: Freitag, 4. August 2017 19:45
To: Michael Johnson
Cc:
Subject: Re: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi Michael,

Thanks for the reply.  We are testing with the new try-before-purchase Dell servers (R430). We have installed the perfSONAR Toolkit (pS-Toolkit-4.X-CentOS7-FullInstall-x86_64.iso) on the boxes.

 

 

 

We constantly have these retransmissions. Based on the results, we are not confident to buy 40 new Dell servers and redeploy the perfSONAR nodes in our network. Because we are not sure it’s a hardware or software problem.  (If we install Centos 6.9 with latest perfSONAR Toolkit, we don’t get these transmissions.)

 

We are really running out of ideas. We are thinking of connecting 2 servers back-to-back to eliminate the network issues and testing again, to see if the retransmission remains.

 

Any suggestions?

 

 

Tao Zhang

 Network Engineer | CANARIE Inc | 45 O'Connor St., Suite 500, Ottawa, ON K1P 1A4 │CANARIE NOC: 613-944-5612│www.canarie.ca

 

 

 

 

From: Michael Johnson <>
Date: Monday, July 31, 2017 at 11:00 AM
To: Tao Zhang <>
Cc: "" <>
Subject: Re: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi Tao,

 

It's true that there are more EL6 hosts out there than EL7. However, most of the development testing is now taking place on CentOS 7, and it is fully supported.

 

I believe the reason you're seeing discrepancies is because CentOS 7 has a different kernel, which sometimes handles things a bit differently. In addition to that, the most recent versions of iperf3 have some relevant changes as well. The end result is that we are getting better at detecting packet retransmissions we weren't seeing before (this is consistent with the behavior we've seen in testing). That's my understanding, anyway.

 

I hope this helps!

 

Thanks,

 

Michael Johnson

GlobalNOC Software Engineering

Indiana University

812-856-2771

 

 

 

 

Hi perfSONAR community,

 

As we are planning to redeploy our perfSONAR nodes, we have done thorough tests among our perfSONAR boxes. All the nodes are running perfSONAR Toolkit v4.0.0.1.

Here are the throughput test results:

 

Source	Destination	Results
Centos6.9	Centos6.9	Clean 10G Path (not have a single retransmission within years)
Centos7.3	Centos7.3	Packet Retransmission (can reach 10G, but have packet retransmission once in a while)
Centos6.9	Centos7.3	Packet Retransmission (can reach 10G, but have packet retransmission once in a while)

 

It looks like the perfSONAR 4.0 doesn’t work very well with Centos 7.  I checked the perfSONAR Lookup Service Directory and found that most nodes are still running on Centos 6.9. I suspect that it may be the reason the problem is not fully exposed.

 

Anyone has similar problems?

 

Thanks

 

 

Tao Zhang

 Network Engineer | CANARIE Inc | 45 O'Connor St., Suite 500, Ottawa, ON K1P 1A4 │CANARIE NOC: 613-944-5612│www.canarie.ca