perfSONAR User Q&A and Other Discussion

[Hyojoon Kim <>]

Hi all, 

After some digging, I am pretty convinced that the under-bandwidth achievement of the LIVA X box is due to:

 - "perfsonar-toolkit-security” package installation. 

“lsmod" command shows that nf_conntrack-related kernel modules become loaded after the perfsonar security package installation, along with several iptables rules. After this happens, bwctl measurement will never go over 700Mbps when the test is initiated from the LIVA X box. Flushing the iptables rules and disabling the nf_conntrack-related modules fixes the issue, restoring the bwctl measurement to 940Mbps. 

Can someone confirm this? 

Of course, you have to be able to remove the security package after testing. You can do:

# apt-get remove perfsonar-toolkit-security
# apt-get autoremove
# cd /etc/iptables/
# rm rules.v4 rules.v6
# reboot 

If this is confirmed, maybe it makes sense to put a note at this link, in the section of “Installation Instructions”.

http://docs.perfsonar.net/low_cost_nodes.html

Or at this link, putting a note saying something like “Don’t install the security package if you have a low-cost node, like LIVA X”. 

http://docs.perfsonar.net/install_debian.html

Happy holidays, everyone.

Thanks,

Joon

On Dec 30, 2015, at 10:20 AM, Hyojoon Kim <> wrote:

Hi Jason, 

Sorry for the late reply; was a bit out for the holidays, and I wanted to actually replicate the problem. 

- tcpdumps when the LIVA box only achieves around 513Mbps is here. Note that if the test is initiated *from* the Dell server, it does achieve 935Mbps. 

https://www.dropbox.com/s/o5lg0ypqddevnjs/tcpdump_perfsonar.tgz?dl=0

- TCP settings are probably not exactly uniform. Let me know which settings you want to see, and I will provide them from "sysctl -a".

- The Dell server has CentOS. 

Thanks,

Joon 

On Dec 24, 2015, at 9:35 AM, Jason Zurawski <> wrote:

Hey Joon;

Interesting stuff - its a little late to weigh in, but some thoughts as I read this:

    - You noted that when the Liva was the receiver (and Dell was sending, things were 'ok'.  The opposite (Dell receiving, Liva sending) things degraded.  Did you happen to capture any packet traces during this?  I would be curious to see if there was something funky going on that would force a slowdown (pause frames, window manipulation, etc.) either from one of the OSs or the NICs. 

    - Were the TCP settings uniform on both ends?  Same congestion control, same socket sizes, etc?

    - Was the Dell a CentOS system, or also an ubuntu?

Glad its 'working', not glad that things are this touchy :)

-jason

Daniel Doyle wrote:

Joon,

Excellent! And thanks for passing along the information. I'll have to dig a bit on this and figure out what the offending bit was, but in the meanwhile I will update the page to reflect these findings in case others are poking around and run into the same issue.

-Dan

On Dec 22, 2015, at 6:24 PM, Hyojoon Kim <> wrote:

Just to give an update on this issue: 

On a different LIVA X 2GB/32GB box, I installed “Ubuntu-12.04.5-desktop-amd64” (http://releases.ubuntu.com/12.04/ubuntu-12.04.5-desktop-amd64.iso) and removed the ondemand stuff (with "update-rc.d -f ondemand remove”).

Now I get 940Mbps with bwctl :-) 

* Side note: 

I don’t know what makes the other LIVA box with Ubuntu-14.04-desktop-amd64 unable to achieve 940Mbps. I might dig into it when I have the time. Just FYI, things I did differently on the under-achieving box are:

  - It’s a different OS (Ubuntu 14.04)

  - I installed two more packages ("perfsonar-toolkit-security" and "perfsonar-toolkit-sysctl") in addition to the "perfsonar-testpoint”

  - I ran the “Software Updater” (https://en.wikipedia.org/wiki/Software_Updater) and updated everything before running bwctl. 

  - I did some Linux tuning after it was not able to achieve 940 Mbps. 

Thanks,

Joon 

On Dec 22, 2015, at 2:00 PM, Hyojoon Kim <> wrote:

Thanks for all the suggestions and comments!

To answer some:
* I did remove the "ondemand option" as I saw that note somewhere too before I ran the test. I’m sure it did something but I still get around 550Mbps bwctl.
* I did the Linux Tuning *after* I got 550Mbps at my first trial, hoping it would fix it. No luck.
* Interestingly, I get around 940Mbps when I initiate the bwctl test *from* a Dell node *to* the LIVA X box. But the other direction still gives me around 550Mbps.
* One thing I noticed is that:
 - When I initiate the test from the Dell node, it opens an ephemeral port (e.g., 45250) on the dell node.
     - local LIVAXBOX port 5593 connected with DELL port 45240
 - From LIVAX box to Dell server is:
     - local DELL port 5220 connected with LIVAXBOX port 5220


Thanks,
Joon

On Dec 22, 2015, at 1:43 PM, Uhl, George D. (GSFC-423.0)[SGT INC] <> wrote:

When I was researching the Liva X capabilities I ran across an email from Larry Blunk of Merit that he posted on the list this past October.

 One thing to note is that Ubuntu enables the "ondemand" init script
by default which puts the CPU in "powersave" mode.   In testing the
LIVA, I found that this seems to limit throughput a bit in performance
tests.  I get around 900Mbps TCP throughput vs. 940Mbps in
"performance" mode.  Also saw some packet loss when doing UDP tests
with 1500 byte datagrams at 1 Gbps in powersave mode.  You can disable
the script with the following which will leave the CPU in performance
mode.

update-rc.d -f ondemand remove



-Larry Blunk
 Merit

Disabling the “ondemand” mode capability might work for you.   I did it and I was able to achieve 940 Mbps.

-George

From: <> on behalf of Brian Tierney <>
Reply-To: "" <>
Date: Tuesday, December 22, 2015 at 1:12 PM
To: Daniel Doyle <>
Cc: Hyojoon Kim <>, "" <>
Subject: Re: [perfsonar-user] Bandwidth measurement with LIVA X 2GB/32GB eMMC



On Tue, Dec 22, 2015 at 12:00 PM, Daniel Doyle <> wrote:

Hi Joon,

Sorry to hear about that. A couple of questions / debugging ideas:

- Are you seeing this in both directions?
- Have you tried using the same port with a different device?
- If you're using / not using jumbo frames has the device been configured accordingly?

fwiw, I did not apply any tunings out of the box on a LIVA X and got >900Mbps on a local network. It's possible some of those tunings might not be appropriate for a machine like that, but I have't dug into it.

I agree that might be the issue. The tunings on fasterdata might not be right for a small node.
See if the original default debian settings are better.

-Dan

On Dec 22, 2015, at 11:58 AM, Hyojoon Kim <> wrote:

Hi all,

We decided to play with a LIVA X 2GB/32GB eMMC node with perfSONAR 3.5. However, I am getting around 535Mbps instead of 940+Mbps when I initiate a bwctl test to another local perfSONAR node we have. Between two Dell R610 servers with perfSONAR, we do get over 950Mbps, so it’s likely not a network problem.

I have generally followed the instructions here (http://docs.perfsonar.net/install_debian.html), and installed three packages:
apt-get install perfsonar-testpoint
apt-get install perfsonar-toolkit-security
apt-get install perfsonar-toolkit-sysctl

I also did the Linux Tuning, which is indicated here:
http://fasterdata.es.net/host-tuning/linux/

Does anyone know what I am missing? I do have Ubuntu 14.04.3 Desktop OS on this. Maybe switching to the well-tested Ubuntu 12.04 is better?

===
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:        14.04
Codename:       trusty

$ uname -a
Linux perfbox-livaxtest-01 3.19.0-39-generic #44~14.04.1-Ubuntu SMP Wed Dec 2 10:00:35 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
===

Thanks,
Joon