perfSONAR User Q&A and Other Discussion

["Roderick Mooi" <>]

Hi Paul, Brian

1. Looking for loss is an obvious oversight - there is indeed some:
see:
https://192.96.2.247/serviceTest/delayGraph.cgi?url=http://localhost:8085/perfSONAR_PS/services/pSB&key=de3625b0c481ef8338aab14be049313d&keyR=2d31fa42a62b9188f88046b2f24b7510&dstIP=192.96.2.247&srcIP=196.21.48.249&dst=192.96.2.247&src=196.21.48.249&type=TCP&length=604800&bucket_width=0.0001

2. There are some dropped packets on the switch interface but this number 
didn't increase on a "bad" run...:

"  Queue counters:       Queued packets  Transmitted packets      Dropped 
packets
    0 best-effort                    0           7740886790               
134859
"

The measurement node is connected to a 1G copper interface on this Juniper 
ex4200-24t...

3. generic-receive-offloading was enabled on one of the NICs (no other 
offloading was enabled). Disabling this didn't seem to make a difference.

4. tcpdump & tcptrace:

This confirmed the drops due to retransmitted packets in a "bad" run:
...
     rexmt data pkts:         885           rexmt data pkts:           0
     rexmt data bytes:    1281480           rexmt data bytes:          0
...
   post-loss acks:          176           post-loss acks:            0
...
     duplicate acks:           92           duplicate acks:            0
     triple dupacks:            7           triple dupacks:            0
     max # retrans:             6           max # retrans:             0

(full output of the connections for a "good" and "bad" run are attached)

Now that we know there is loss and retransmitted packets do you have 
suggestions on what to do next? I'm still puzzled by the fact that all tests 
between these 2 nodes and others nodes on the network path are fine (i.e. I 
don't see this up-down behaviour).
[see:
http://perfsonara.sanren.ac.za/serviceTest/index.cgi?eventType=bwctl
and
https://192.96.2.247/serviceTest/index.cgi?eventType=bwctl
]

Thanks!

Roderick

>>> On 2013-10-16 at 18:26, "Wefel, Paul" 
>>> <>
>>>  wrote:
> Couple ideas
> 
> Run owamp between these two hosts looking for packet loss in only one
> direction.
> Check the switch interface that Dst is connected to looking for queue
> drops and pause frames being sent.
> 
> I have also seen strange issues with some NICS when offloading is enabled
> on them.
> 
> good luck, let us know what you find.
> 
> -paul
> NCSA @ UIUC
> 
> -----Original Message-----
> From: Roderick Mooi 
> <>
> Date: Wednesday, October 16, 2013 5:07 AM
> To: 
> ""
> <>,
>  
> ""
> <>
> Subject: [perfsonar-user] Help with inconsistent bwctl measurements
> 
>>Hi
>>
>>I have been trying to locate the cause of inconsistent measurements
>>between two nodes for a few weeks now without success. The pattern I'm
>>seeing is available at:
>>
>>https://192.96.2.247/serviceTest/bandwidthGraph.cgi?url=http://localhost:8 
>>085/perfSONAR_PS/services/pSB&key=d9013ce7df20b8bbe45defeaeae785d6&keyR=0a
>>0ed6c928edf28976414a2cc7e87d6f&dstIP=192.96.2.247&srcIP=196.21.48.249&dst=
>>192.96.2.247&src=perfsonara.sanren.ac.za&type=TCP&length=2592000
>>
>>Src-Dst is consistent but Dst-Src is not.
>>
>>Manual tests (attached) show the same behaviour without any indication of
>>cause - measures 941 Mbps then drops to 189 Mbps (end) and back to 941
>>(nothing different in the logs between "good" measurements and "bad"
>>ones). The only time I've seen something similar is when I was testing
>>from a 10 G interface to a 1 G interface which was subsequently being
>>flooded. In this case both interfaces are 1 G. I'm also not seeing any
>>problems with measurements along the path or between these nodes and any
>>other nodes. Additionally, there is very little (< 50 Mbps) real traffic
>>between these 2 nodes.
>>
>>Any ideas?
>>
>>Thanks!
>>
>>Roderick
>>
>>-- 
>>Roderick Mooi      | SANREN Engineer
>>-- 
>>
>> | +27 12 841 4111 | www.sanren.ac.za 
>> 
>>
>>
>>
>>-- 
>>This message is subject to the CSIR's copyright terms and conditions,
>>e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>The full disclaimer details can be found at
>>http://www.csir.co.za/disclaimer.html.
>>
>>This message has been scanned for viruses and dangerous content by
>>MailScanner, 
>>and is believed to be clean.
>>
>>Please consider the environment before printing this email.
>>
> 
> 
> -- 
> This message is subject to the CSIR's copyright terms and conditions, 
> e-mail 
> legal notice, and implemented Open Document Format (ODF) standard. 
> The full disclaimer details can be found at 
> http://www.csir.co.za/disclaimer.html.
> 
> This message has been scanned for viruses and dangerous content by 
> MailScanner, 
> and is believed to be clean.
> 
> Please consider the environment before printing this email.

-- 
This message is subject to the CSIR's copyright terms and conditions, e-mail 
legal notice, and implemented Open Document Format (ODF) standard. 
The full disclaimer details can be found at 
http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by 
MailScanner, 
and is believed to be clean.

Please consider the environment before printing this email.