Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Help with inconsistent bwctl measurements

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Help with inconsistent bwctl measurements


Chronological Thread 
  • From: "Roderick Mooi" <>
  • To: "Brian Tierney" <>, "Paul Wefel" <>
  • Cc: "Kevin Draai" <>, "" <>, "" <>
  • Subject: Re: [perfsonar-user] Help with inconsistent bwctl measurements
  • Date: Thu, 17 Oct 2013 10:01:34 +0200

Hi Paul, Brian

1. Looking for loss is an obvious oversight - there is indeed some:
see:
https://192.96.2.247/serviceTest/delayGraph.cgi?url=http://localhost:8085/perfSONAR_PS/services/pSB&key=de3625b0c481ef8338aab14be049313d&keyR=2d31fa42a62b9188f88046b2f24b7510&dstIP=192.96.2.247&srcIP=196.21.48.249&dst=192.96.2.247&src=196.21.48.249&type=TCP&length=604800&bucket_width=0.0001

2. There are some dropped packets on the switch interface but this number
didn't increase on a "bad" run...:

" Queue counters: Queued packets Transmitted packets Dropped
packets
0 best-effort 0 7740886790
134859
"

The measurement node is connected to a 1G copper interface on this Juniper
ex4200-24t...

3. generic-receive-offloading was enabled on one of the NICs (no other
offloading was enabled). Disabling this didn't seem to make a difference.

4. tcpdump & tcptrace:

This confirmed the drops due to retransmitted packets in a "bad" run:
...
rexmt data pkts: 885 rexmt data pkts: 0
rexmt data bytes: 1281480 rexmt data bytes: 0
...
post-loss acks: 176 post-loss acks: 0
...
duplicate acks: 92 duplicate acks: 0
triple dupacks: 7 triple dupacks: 0
max # retrans: 6 max # retrans: 0

(full output of the connections for a "good" and "bad" run are attached)

Now that we know there is loss and retransmitted packets do you have
suggestions on what to do next? I'm still puzzled by the fact that all tests
between these 2 nodes and others nodes on the network path are fine (i.e. I
don't see this up-down behaviour).
[see:
http://perfsonara.sanren.ac.za/serviceTest/index.cgi?eventType=bwctl
and
https://192.96.2.247/serviceTest/index.cgi?eventType=bwctl
]

Thanks!

Roderick

>>> On 2013-10-16 at 18:26, "Wefel, Paul"
>>> <>
>>> wrote:
> Couple ideas
>
> Run owamp between these two hosts looking for packet loss in only one
> direction.
> Check the switch interface that Dst is connected to looking for queue
> drops and pause frames being sent.
>
> I have also seen strange issues with some NICS when offloading is enabled
> on them.
>
> good luck, let us know what you find.
>
> -paul
> NCSA @ UIUC
>
> -----Original Message-----
> From: Roderick Mooi
> <>
> Date: Wednesday, October 16, 2013 5:07 AM
> To:
> ""
> <>,
>
> ""
> <>
> Subject: [perfsonar-user] Help with inconsistent bwctl measurements
>
>>Hi
>>
>>I have been trying to locate the cause of inconsistent measurements
>>between two nodes for a few weeks now without success. The pattern I'm
>>seeing is available at:
>>
>>https://192.96.2.247/serviceTest/bandwidthGraph.cgi?url=http://localhost:8
>>085/perfSONAR_PS/services/pSB&key=d9013ce7df20b8bbe45defeaeae785d6&keyR=0a
>>0ed6c928edf28976414a2cc7e87d6f&dstIP=192.96.2.247&srcIP=196.21.48.249&dst=
>>192.96.2.247&src=perfsonara.sanren.ac.za&type=TCP&length=2592000
>>
>>Src-Dst is consistent but Dst-Src is not.
>>
>>Manual tests (attached) show the same behaviour without any indication of
>>cause - measures 941 Mbps then drops to 189 Mbps (end) and back to 941
>>(nothing different in the logs between "good" measurements and "bad"
>>ones). The only time I've seen something similar is when I was testing
>>from a 10 G interface to a 1 G interface which was subsequently being
>>flooded. In this case both interfaces are 1 G. I'm also not seeing any
>>problems with measurements along the path or between these nodes and any
>>other nodes. Additionally, there is very little (< 50 Mbps) real traffic
>>between these 2 nodes.
>>
>>Any ideas?
>>
>>Thanks!
>>
>>Roderick
>>
>>--
>>Roderick Mooi | SANREN Engineer
>>--
>>
>> | +27 12 841 4111 | www.sanren.ac.za
>>
>>
>>
>>
>>--
>>This message is subject to the CSIR's copyright terms and conditions,
>>e-mail legal notice, and implemented Open Document Format (ODF) standard.
>>The full disclaimer details can be found at
>>http://www.csir.co.za/disclaimer.html.
>>
>>This message has been scanned for viruses and dangerous content by
>>MailScanner,
>>and is believed to be clean.
>>
>>Please consider the environment before printing this email.
>>
>
>
> --
> This message is subject to the CSIR's copyright terms and conditions,
> e-mail
> legal notice, and implemented Open Document Format (ODF) standard.
> The full disclaimer details can be found at
> http://www.csir.co.za/disclaimer.html.
>
> This message has been scanned for viruses and dangerous content by
> MailScanner,
> and is believed to be clean.
>
> Please consider the environment before printing this email.

--
This message is subject to the CSIR's copyright terms and conditions, e-mail
legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at
http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by
MailScanner,
and is believed to be clean.

Please consider the environment before printing this email.




Archive powered by MHonArc 2.6.16.

Top of Page