perfSONAR User Q&A and Other Discussion

[Alan Whinery <>]

I wrote more below than is perhaps advisable, but I've been living the
Little-Node lifestyle, and I keep seeing conversations where something
I've encountered might be relevant --

I am aiming more for $50 to $60 deployed, $120 is too expensive.

Although many have been looking at special kernels, I haven't seen the
need. I'm deploying small nodes with bwctls 1.5 (notable for its
inclusion of owping/ping/traceroute/tracepath tests, in addition to
iperf/nuttcp), and owampd.

Something I've made use of in the past, where some other org primarily
administered a machine, but allowed me to install owampd/bwctld on it,
was that given a toolkit node at one end, tests to what I would call a
"sensor" machine can be scheduled on the toolkit node without worrying
about all the other stuff that we call perfSONAR on the sensor node.

Just this week, I have 10 BeagleBone Blacks positioned in "user
perspective" positions on our campus, with a project target of 100
nodes. Currently, the BeagleBone Black supply is limited. Several
outlets that I monitor are limiting sales to one-per-customer. I may be
forced to continue with RPI's which doesn't really bother me; CAIDA has
pretty good results with them, and if there are problems, it gives me
something more to write about for the CC-NIE report.

I have 4 CuBox-i4-Pro on order, simply to be able to compare and
contrast the platform with GigE. They are supposed to ship May 15. We'll
see. I was considering ordering a bunch of the low-end models which are
more like BBB in features, but their shipping price reduces until you
get to about $5 per unit at 4 units, and then jumps up to $10 per unit
permanently. Apparently the source is in Israel, unless your country has
a local distributor, which the US does not, but the UK does. It may be
that the $10 doesn't really matter, but it irked me.

There is a new BeagleBone model coming out soon, which is
en/dis-couraging, it has more on-board flash, but every new generation
of BB and CuBox and etc might prioritize things we care less about and
play down the things we do.

The thing I really like about the Black is the internal flash, and I've
created an automatic OS install procedure that notifies status with the
on-board LEDs so one doesn't need to use monitor/keyboard.

There's a lot of speculation about the effects of the "real" Ethernet on
BBB versus internal USB Ethernet on Raspberry Pi, but after fiddling
with both, I have the general impression that it doesn't matter. The
granularity of the USB-introduced jitter/non-determinism versus the
requirements for owping and iperf may be such that the effects are
negligible. I have a BBB, BB(v1), and 3 RPi's laying here, I really need
to pull together a crucible test to see what can be seen about their
functional/operational differences. I'd be interested in anyone's
comments that are more than speculative.

For BB I am using stock kernels, as come with Ubuntu Saucy on
http://www.armhf.com/, I haven't felt a need to reach for web100
kernels. Still, Mark Foster has compiled a Web10G kernel on CuBox (right
on the device), and although there is value to the cross-compile
approach, I would encourage anyone compiling to try an on-device build.
The worst it can do is run for 6 hours and fail, and then you can
continue the cross-compile thing.

One important element of the scale-management is that each BBB comes up
on first boot and registers itself with a Puppet Master
(puppetlabs.com). I am using Puppet as installed by The Foreman
installer (theforeman.org) on a CentOS 6 VM. I have focused a lot of
energy on identical nodes with as close to zero per-device management as
possible, still some ways to go to zero. The BBB nodes prefer IPv6 for
all Puppet communications. I also intend to get PS-SimpleLS server going
there, but I haven't yet needed it, as Puppet/Foreman accounts for
inventory.

I built bwctl and owamp on a BBB and packaged them in Deb files. I have
also dabbled with doing installs on Ubuntu with RPMs, as some of the
pivotal Perl work for perfSONAR is in Perl and therefore ".noarch" but I
am preserving some energy until after the realization of Toolkit 3.4 and
esmond/simpleLS/REST-MA.

Although searching Google for info about binary compatibility for a
binary compiled on one ARM device to another, returns discouragement, a
very simple C program seemed to run OK, but I ran into errors related to
libc version going from amrhf Ubuntu 13.10 to Rasbian Wheezy when I
attempted to install my bwctl .deb files on RPi. So that question is
unresolved, until I can get a close-enough libc match between distros.

As a sidebar -- our wireless guys have been using web100clt on OpenWRT
devices for several years, to test against a perfSONAR toolkit node. So
no web100 kernels on the OpenWRT devices, as they are only clients. I
have provided them with some RPi hardware and a WiFi USB stick to work
on a new generation of this.

Other sidebar -- a recurring question about using PoE comes up, the best
answer I have found so far is probably this:

http://www.amazon.com/HCP05-Passive-Injector-Splitter-Connector/dp/B00DZLSRJC/ref=sr_1_3?ie=UTF8&qid=1398463641&sr=8-3&keywords=poe+splitter

Which does in fact use the same power connector size as BeagleBone. It
would be interesting to look into the hardware docs to see whether the
BB could eat a little over-voltage, so that one could perhaps recover
some voltage drop. For my current project, my devices are close to power
strips, but I could see things like the wireless scenario that it might
be good to get a PoE-LIKE scheme working.

See also:

http://perfclub.org/?p=69

I am happy to make corrections and enhancements, and probably even let
any of you post, if you have something remotely on-topic...

-Alan

On 5/1/2014 9:05 AM, Cort Buffington wrote:
> Folks,
>
> I’m trying hard to let Casey be the “interface” for KanREN’s efforts here — 
> but on this topic, it’s probably easiest if I throw in directly.
>
> I have a BeagleBone Black that I’ve been hoping to use. I’ve been slowly 
> working on a cross-compilation environment to try and build a kernel with 
> the right patches. Do the tests we’re talking about doing with these boxes 
> need that? If not, I’ll skip over that part and star installing the 
> software you have worked on.
>
> We’d targeted the Cubox due to more “power”, but the BeagleBone black is 
> less expensive, more readily available, and has a “real” Ethernet interface 
> rather than the USB-Ethernet of the Raspberry Pi… and I have a BBB, so it 
> seemed like a good place to work in advance of Casey getting our 
> Cubox-i4-Pros.
>
> On May 1, 2014, at 12:39 PM, Brian Tierney 
> <>
>  wrote:
>
>> On Thu, May 1, 2014 at 10:24 AM, Casey Russell 
>> <>
>>  wrote:
>>> Thank you Brian, for all your time in this early testing.
>>>
>>> I'm still waiting for my Cubox-i4-Pro to arrive.  KanREN will be working 
>>> to
>>> test-pilot these devices as soon as we can get our hands on one.
>>>
>>> I'm growing a bit concerned about Solid-Run's ability to supply a decent
>>> number of these things.  I hope they can ramp up production soon, because
>>> these little guys appear to beat the pants off of most other devices in 
>>> the
>>> class.
>> This is a valid concern. Our order took about 3 months to arrive.
>>
>> Has anyone tried using one of these:
>> http://beagleboard.org/Products/BeagleBone
>>
>> I think the same fedora image is supposed to work with that too.
>>
>>
>>> On Thu, May 1, 2014 at 11:40 AM, Brian Tierney 
>>> <>
>>>  wrote:
>>>> Hi Folks:
>>>>
>>>> I've made some progress on getting perfSONAR running on my $120
>>>> cubox-i4Pro.
>>>>
>>>> Attempt #1 was to use their debian kernel and install everything from
>>>> source. That worked fine, but since perfSONAR is RPM-based, I wanted
>>>> to see if I could find an OS for it that supported RPMs.
>>>>
>>>> Then I found a Fedora kernel here:
>>>> http://cubox-i.cf/files/fedora20-kernel3.14rc4-cubox-i-20140310.img.zip
>>>>
>>>> Which works great, except it does not seem to work with an HDMI
>>>> monitor, so you have to put it on a subnet with DHCP and ssh in to
>>>> change the default root password of 'fedora'.
>>>>
>>>> I built some RPMs for bwctl and owamp and temporarily put them here if
>>>> anyone has one of these devices and wants to play with perfSONAR on
>>>> it:
>>>>  http://antg-dev.es.net/rpms/arm/
>>>>
>>>> After these are tested more we can move these to the main Internet2 RPM
>>>> repo.
>>>> And 'yum install nuttcp iperf3 iperf' just works, as all are in the
>>>> main fedora arm repo. Cool!
>>>>
>>>> So far everything is working well.  NTP drift is around 1ms, which is
>>>> usable, and TCP flows max out at 350Mbps, which is good enough for
>>>> many use cases. But I'd recommend mainly using these devices for
>>>> packet loss testing, not throughput testing.
>>>>
>>>>
>>>>
>>>> --
>>>> Brian Tierney, http://www.es.net/tierney
>>>>
>>>> Energy Sciences Network (ESnet), Berkeley National Lab
>>>>
>>>> http://fasterdata.es.net
>>>
>>
>>
>> -- 
>> Brian Tierney, http://www.es.net/tierney
>>
>> Energy Sciences Network (ESnet), Berkeley National Lab
>>
>> http://fasterdata.es.net
> --
> Cortney T. Buffington
> Executive Director
> KanREN, Inc.
> 
> Office: (785) 856-9800
> Mobile: (785) 865-7206
>