perfSONAR User Q&A and Other Discussion

[Thomas Tam <>]

Hi All,

 

Thanks for those who has responded to the Tao’s post.

 

I just want to give a context of why we are concerning the retransmission on Centos 7. We are looking into replacing all of our old perfSONAR servers, planning to buy around 30 of new servers. if these retransmissions are related to an inadequacy of the server hardware, we want to know before buying them. If this issue has been observed from a number of perfSONAR administrators after upgrading to Centos 7, perhaps this has nothing to do with the hardware specification. We would be comfortable to go ahead with the purchase without knowing the causes.

 

We have spent considerable amount of time to try various configurations. The latest test was done between two boxes connected back-to-back. This should eliminate the issues of packet loss, out-of-order and lack of memory buffer, but we were still seeing retransmissions. Honestly, we aren’t experts in perfSONAR and Centos, we relate on the expertise in the community helping us to make a right choice.

 

Your feedbacks are grealy appericated.

 

Best regards,

-Thomas

 

 

From: Tao Zhang <>
Date: Wednesday, August 23, 2017 at 8:52 AM
To: "Garnizov, Ivan (RRZE)" <>, Michael Johnson <>
Cc: "" <>
Subject: Re: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi All,

I understand pS is a measurement framework, not a measurement tool.  We have tried many test scenarios (different types of servers, different system parameter settings, different 10G NIC cards) trying to figure out why perfSONAR (iperf3) throughput tests have these TCP retransmissions on Centos 7.

Does anyone who is running perfSONAR on Centos 7 and not having these TCP retransmissions?

 

Thanks

 

Tao

 

 

From: "Garnizov, Ivan (RRZE)" <>
Date: Monday, August 7, 2017 at 6:49 AM
To: Tao Zhang <>, Michael Johnson <>
Cc: "" <>
Subject: RE: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi Tao,

 

IMHO there is a bit of a misunderstanding here. perfSONAR toolkit is not a measurement tool. pS is a measurement framework, which can coordinate measurements of multiple measurement tools and much more.

 

perfSONAR toolkit indeed provides some kernel and interface optimizations out of the box, but these are still generic and there is no such one-size-fits-all solution. After all you are reaching 10G.

 

You will have to go in the details and reach for the respective tool maintainers (depending on the tool you use) to get a clear understanding on the discrepancy you observe.

The difference between CentOS 6.9 and CentOS 7 is not only about the kernel. It is also about the drivers of the NIC.

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

 

 

From: [mailto:] On Behalf Of Tao Zhang
Sent: Freitag, 4. August 2017 19:45
To: Michael Johnson
Cc:
Subject: Re: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi Michael,

Thanks for the reply.  We are testing with the new try-before-purchase Dell servers (R430). We have installed the perfSONAR Toolkit (pS-Toolkit-4.X-CentOS7-FullInstall-x86_64.iso) on the boxes.

 

 

 

We constantly have these retransmissions. Based on the results, we are not confident to buy 40 new Dell servers and redeploy the perfSONAR nodes in our network. Because we are not sure it’s a hardware or software problem.  (If we install Centos 6.9 with latest perfSONAR Toolkit, we don’t get these transmissions.)

 

We are really running out of ideas. We are thinking of connecting 2 servers back-to-back to eliminate the network issues and testing again, to see if the retransmission remains.

 

Any suggestions?

 

 

Tao Zhang

 Network Engineer | CANARIE Inc | 45 O'Connor St., Suite 500, Ottawa, ON K1P 1A4 │CANARIE NOC: 613-944-5612│www.canarie.ca

 

 

 

 

From: Michael Johnson <>
Date: Monday, July 31, 2017 at 11:00 AM
To: Tao Zhang <>
Cc: "" <>
Subject: Re: [perfsonar-user] Packet Retransmission on Centos 7

 

Hi Tao,

 

It's true that there are more EL6 hosts out there than EL7. However, most of the development testing is now taking place on CentOS 7, and it is fully supported.

 

I believe the reason you're seeing discrepancies is because CentOS 7 has a different kernel, which sometimes handles things a bit differently. In addition to that, the most recent versions of iperf3 have some relevant changes as well. The end result is that we are getting better at detecting packet retransmissions we weren't seeing before (this is consistent with the behavior we've seen in testing). That's my understanding, anyway.

 

I hope this helps!

 

Thanks,

 

Michael Johnson

GlobalNOC Software Engineering

Indiana University

812-856-2771

 

 

 

 

Hi perfSONAR community,

 

As we are planning to redeploy our perfSONAR nodes, we have done thorough tests among our perfSONAR boxes. All the nodes are running perfSONAR Toolkit v4.0.0.1.

Here are the throughput test results:

 

Source	Destination	Results
Centos6.9	Centos6.9	Clean 10G Path (not have a single retransmission within years)
Centos7.3	Centos7.3	Packet Retransmission (can reach 10G, but have packet retransmission once in a while)
Centos6.9	Centos7.3	Packet Retransmission (can reach 10G, but have packet retransmission once in a while)

 

It looks like the perfSONAR 4.0 doesn’t work very well with Centos 7.  I checked the perfSONAR Lookup Service Directory and found that most nodes are still running on Centos 6.9. I suspect that it may be the reason the problem is not fully exposed.

 

Anyone has similar problems?

 

Thanks

 

 

Tao Zhang

 Network Engineer | CANARIE Inc | 45 O'Connor St., Suite 500, Ottawa, ON K1P 1A4 │CANARIE NOC: 613-944-5612│www.canarie.ca