perfSONAR User Q&A and Other Discussion

["Wu, Xiaoban" <>]

Dear Eli,

Thanks so much for your reply. But I still have some questions. 

First, for the graph in the link http://fasterdata.es.net/network-tuning/tcp-issues-explained/packet-loss/ , we can see that there is a steep drop from 0ms to 10ms except for the pink plot, but it says that "TCP can still perform OK over such distances", I want to know if this is a contradiction? If not, could you please explain why in more details? Thanks so much.

Second, it says that " This makes sense because in ESnet we monitor our own network internally and have a high degree of confidence that it is functioning correctly." What if unluckily it is exactly the ESnet's problem instead of the other networks? (Because, If everybody is confident in their own network, then we maybe don't know where to start to find the problems) Then it says "we immediately have something to investigate within ESnet", I want to know in detail how you could find the problem within such a small distance like 30ms as in the graph on page 20, since "TCP can still perform OK over such distances".                                              

Third, it says " try starting at the other end of the path", what if unfortunately the other end also has the problem? In this case, how could we solve this issue? Thanks so much.

All the best,

Xiaoban

 

---

From: Eli Dart <>
Sent: Thursday, August 27, 2015 6:44 PM
To: Wu, Xiaoban
Cc:
Subject: Re: [perfsonar-user] Some questions about a slide

 

Hi Xiaoban,

Segment by segment testing has the potential of showing high performance even when there are problems. To see why, take a look at the TCP loss graph here:

http://fasterdata.es.net/network-tuning/tcp-issues-explained/packet-loss/

When the latency is low, performance can still be good even in the presence of packet loss.

Note that in saying "segment by segment testing" we mean something like the following. Label each perfSONAR host in the path, in order, 1, 2, 3, 4, 5, ... N. Then test from 1-2, from 2-3, from 3-4, and so on. This method is flawed, because the maximum latency of a test is governed by the length of any one segment. It is possible that some segments are long and others are short, but in most networks the segments will be short enough that the latency will be 10 milliseconds or less.  Depending on the amount of loss, TCP can still perform OK over such distances, thus making it difficult to identify the problem segment using this testing methodology.

So, I find it's better to pick a test host as an anchor and gradually increase the length of the test. 

Looking at the diagram on slide 20, and picking the ESnet perfSONAR host closest to the national laboratory (dark green) as an anchor point, we might test first to the ESnet perfSONAR host closest to the handoff to Internet2. This makes sense because in ESnet we monitor our own network internally and have a high degree of confidence that it is functioning correctly. If performance between the two ESnet testers is not what it should be, then we immediately have something to investigate within ESnet.

We would then test to the closest Internet2 perfSONAR host on the other side of the ESnet/Internet2 peering.  If that's good, then test to the far Internet2 perfSONAR host (the one closest to the peering between Internet2 and the purple regional network).

Then, test to the perfSONAR host in the regional network just beyond the peering with Internet2. And so on.

If performance goes down significantly as a function of distance as you test progressively further, you may have a problem close to your anchor tester. If that's the case, try starting at the other end of the path.

The way you know the long path is clean is that a TCP BWCTL test performs well.  If there is loss, it will perform poorly.  In the real-world testing from which these diagrams were derived, the difference between the performance of the clean test and the dirty test was dramatic - it's not a subtle thing.

Does this help?

Thanks,

Eli

On Thu, Aug 27, 2015 at 2:16 PM, Wu, Xiaoban <> wrote:

Dear All,

I have been reading this slide http://www.perfsonar.net/media/cms_page_media/3271/20150701-perfSONAR-7-Debugging_Strategies-v1.pdf  , but now I am struggling with some questions. 

On page 10 and page 17, it says we need to find the longest clean path and segment-to-segment test is not helpful. I want to know if anybody has done some tests to verify that segment-to-segment test is not helpful. If so, could you please provide the details of how you implement this test? Thanks so much for your help.

On the other hand, on page 20, it shows a long clean path. I want to know how to make sure that it is clean, so we can trust it. Moreover, how to find such long clean path? Since segment-to-segment test is not helpful, gradually increasing the path probably may deceive us. If you have done any similar test, could you please also provide some details? Thanks so much for your help.

All the best,

Xiaoban

--

Eli Dart, Network Engineer                          NOC: (510) 486-7600

ESnet Office of the CTO (AS293)                          (800) 333-7638

Lawrence Berkeley National Laboratory 

PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3