perfSONAR User Q&A and Other Discussion

[Daniel Doyle <daldoyle@grnoc.iu.edu>]

Joon,

Excellent! And thanks for passing along the information. I'll have to dig a bit on this and figure out what the offending bit was, but in the meanwhile I will update the page to reflect these findings in case others are poking around and run into the same issue.

-Dan

On Dec 22, 2015, at 6:24 PM, Hyojoon Kim <joonk@princeton.edu> wrote:

Just to give an update on this issue: 

On a different LIVA X 2GB/32GB box, I installed “Ubuntu-12.04.5-desktop-amd64” (http://releases.ubuntu.com/12.04/ubuntu-12.04.5-desktop-amd64.iso) and removed the ondemand stuff (with "update-rc.d -f ondemand remove”).

Now I get 940Mbps with bwctl :-) 

* Side note: 

I don’t know what makes the other LIVA box with Ubuntu-14.04-desktop-amd64 unable to achieve 940Mbps. I might dig into it when I have the time. Just FYI, things I did differently on the under-achieving box are:

  - It’s a different OS (Ubuntu 14.04)

  - I installed two more packages ("perfsonar-toolkit-security" and "perfsonar-toolkit-sysctl") in addition to the "perfsonar-testpoint”

  - I ran the “Software Updater” (https://en.wikipedia.org/wiki/Software_Updater) and updated everything before running bwctl. 

  - I did some Linux tuning after it was not able to achieve 940 Mbps. 

Thanks,

Joon 

On Dec 22, 2015, at 2:00 PM, Hyojoon Kim <joonk@princeton.edu> wrote:

Thanks for all the suggestions and comments!

To answer some:
* I did remove the "ondemand option" as I saw that note somewhere too before I ran the test. I’m sure it did something but I still get around 550Mbps bwctl.
* I did the Linux Tuning *after* I got 550Mbps at my first trial, hoping it would fix it. No luck.
* Interestingly, I get around 940Mbps when I initiate the bwctl test *from* a Dell node *to* the LIVA X box. But the other direction still gives me around 550Mbps.
* One thing I noticed is that:
 - When I initiate the test from the Dell node, it opens an ephemeral port (e.g., 45250) on the dell node.
     - local LIVAXBOX port 5593 connected with DELL port 45240
 - From LIVAX box to Dell server is:
     - local DELL port 5220 connected with LIVAXBOX port 5220


Thanks,
Joon

On Dec 22, 2015, at 1:43 PM, Uhl, George D. (GSFC-423.0)[SGT INC] <george.d.uhl@nasa.gov> wrote:

When I was researching the Liva X capabilities I ran across an email from Larry Blunk of Merit that he posted on the list this past October.

 One thing to note is that Ubuntu enables the "ondemand" init script
by default which puts the CPU in "powersave" mode.   In testing the
LIVA, I found that this seems to limit throughput a bit in performance
tests.  I get around 900Mbps TCP throughput vs. 940Mbps in
"performance" mode.  Also saw some packet loss when doing UDP tests
with 1500 byte datagrams at 1 Gbps in powersave mode.  You can disable
the script with the following which will leave the CPU in performance
mode.

update-rc.d -f ondemand remove



-Larry Blunk
 Merit

Disabling the “ondemand” mode capability might work for you.   I did it and I was able to achieve 940 Mbps.

-George

From: <perfsonar-user-request@internet2.edu> on behalf of Brian Tierney <bltierney@es.net>
Reply-To: "bltierney@es.net" <bltierney@es.net>
Date: Tuesday, December 22, 2015 at 1:12 PM
To: Daniel Doyle <daldoyle@grnoc.iu.edu>
Cc: Hyojoon Kim <joonk@princeton.edu>, "perfsonar-user@internet2.edu" <perfsonar-user@internet2.edu>
Subject: Re: [perfsonar-user] Bandwidth measurement with LIVA X 2GB/32GB eMMC



On Tue, Dec 22, 2015 at 12:00 PM, Daniel Doyle <daldoyle@grnoc.iu.edu> wrote:

Hi Joon,

Sorry to hear about that. A couple of questions / debugging ideas:

- Are you seeing this in both directions?
- Have you tried using the same port with a different device?
- If you're using / not using jumbo frames has the device been configured accordingly?

fwiw, I did not apply any tunings out of the box on a LIVA X and got >900Mbps on a local network. It's possible some of those tunings might not be appropriate for a machine like that, but I have't dug into it.

I agree that might be the issue. The tunings on fasterdata might not be right for a small node.
See if the original default debian settings are better.

-Dan

On Dec 22, 2015, at 11:58 AM, Hyojoon Kim <joonk@princeton.edu> wrote:

Hi all,

We decided to play with a LIVA X 2GB/32GB eMMC node with perfSONAR 3.5. However, I am getting around 535Mbps instead of 940+Mbps when I initiate a bwctl test to another local perfSONAR node we have. Between two Dell R610 servers with perfSONAR, we do get over 950Mbps, so it’s likely not a network problem.

I have generally followed the instructions here (http://docs.perfsonar.net/install_debian.html), and installed three packages:
apt-get install perfsonar-testpoint
apt-get install perfsonar-toolkit-security
apt-get install perfsonar-toolkit-sysctl

I also did the Linux Tuning, which is indicated here:
http://fasterdata.es.net/host-tuning/linux/

Does anyone know what I am missing? I do have Ubuntu 14.04.3 Desktop OS on this. Maybe switching to the well-tested Ubuntu 12.04 is better?

===
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:        14.04
Codename:       trusty

$ uname -a
Linux perfbox-livaxtest-01 3.19.0-39-generic #44~14.04.1-Ubuntu SMP Wed Dec 2 10:00:35 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
===

Thanks,
Joon

Dan Doyle
GlobalNOC Software Developer
1-812-856-3892
daldoyle@grnoc.iu.edu

--
Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net

Dan Doyle

GlobalNOC Software Developer

1-812-856-3892

daldoyle@grnoc.iu.edu

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail