perfsonar-user - Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi

From: Gaetano Vocca <>
To: Aaron Brown <>
Cc: Alan Whinery <>, "" <>
Subject: Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi
Date: Fri, 19 Dec 2014 21:20:27 +0100

Hi Aaron,

please find attached the file containing the text which is passed by stdin to a Perl script I use as post hook.

I detect a problem in the RECV_OUTPUT section, line 169.

You will also notice that there is an error at line 178 which doesn’t seem to affect the test result. I understand that the hooks are fed by the stdin, would it be possible to send these error notifications to stderr?

Thanks again

Gaetano

<TEST_CONFIG>
tool: iperf3
user:
limit_class: root
start_time: 1419020050
is_host_sender: NO
tos: 0
client: raspberrypi.home.net
sender: 192.168.129.94
receiver: raspberrypi.home.net
duration: 2
use_udp: YES
bandwidth: 5000000
window: 0
len_buffer: 0
report_interval: 0
parallel_streams: 0
units:
output_format: J
use_dynamic_window_sizing: NO
</TEST_CONFIG>
<RECV_OUTPUT>
{
"start": {
"connected": [{
"socket": 16,
"local_host": "192.168.129.81",
"local_port": 5665,
"remote_host": "192.168.129.94",
"remote_port": 42630
}],
"version": "iperf 3.0.10",
"system_info": "Linux raspberrypi 3.12.28+ #709 PREEMPT Mon
Sep 8 15:28:00 BST 2014 armv6l GNU/Linux\n",
"timestamp": {
"time": "Fri, 19 Dec 2014 20:14:14 GMT",
"timesecs": 1419020054
},
"accepted_connection": {
"host": "192.168.129.94",
"port": 41644
},
"cookie": "osboxes.1419020055.082847.65ec9a0b4c",
"test_start": {
"protocol": "UDP",
"num_streams": 1,
"blksize": 8192,
"omit": 0,
"duration": 2,
"bytes": 0,
"blocks": 0,
"reverse": 0
}
},
"intervals": [{
"streams": [{
"socket": 16,
"start": 0,
"end": 1.00023,
"seconds": 1.00023,
"bytes": 565248,
"bits_per_second": 4.52094e+06,
"jitter_ms": 4.57404,
"lost_packets": 0,
"packets": 69,
"lost_percent": 0,
"omitted": false
}],
"sum": {
"start": 0,
"end": 1.00023,
"seconds": 1.00023,
"bytes": 565248,
"bits_per_second": 4.52094e+06,
"jitter_ms": 4.57404,
"lost_packets": 0,
"packets": 69,
"lost_percent": 0,
"omitted": false
}
}, {
"streams": [{
"socket": 16,
"start": 1.00023,
"end": 2.00021,
"seconds": 0.999982,
"bytes": 630784,
"bits_per_second": 5.04636e+06,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 77,
"lost_percent": 0,
"omitted": false
}],
"sum": {
"start": 1.00023,
"end": 2.00021,
"seconds": 0.999982,
"bytes": 630784,
"bits_per_second": 5.04636e+06,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 77,
"lost_percent": 0,
"omitted": false
}
}, {
"streams": [{
"socket": 16,
"start": 2.00021,
"end": 2.03238,
"seconds": 0.0321651,
"bytes": 0,
"bits_per_second": 0,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 0,
"lost_percent": nan,
"omitted": false
}],
"sum": {
"start": 2.00021,
"end": 2.03238,
"seconds": 0.0321651,
"bytes": 0,
"bits_per_second": 0,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 0,
"lost_percent": nan,
"omitted": false
}
}],
"end": {
"streams": [{
"udp": {
"socket": 16,
"start": 0,
"end": 2.03238,
"seconds": 2.03238,
"bytes": 1196032,
"bits_per_second": 4.70791e+06,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 146,
"lost_percent": 0
}
}],
"sum": {
"start": 0,
"end": 2.03238,
"seconds": 2.03238,
"bytes": 1196032,
"bits_per_second": 4.70791e+06,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 146,
"lost_percent": 0
},
"cpu_utilization_percent": {
"host_total": 1.61363,
"host_user": 0,
"host_system": 1.61363,
"remote_total": 4.32541,
"remote_user": 0.200171,
"remote_system": 4.30456
}
}
}
{
"start": {
"connected": [],
"version": "iperf 3.0.10",
"system_info": "Linux raspberrypi 3.12.28+ #709 PREEMPT Mon
Sep 8 15:28:00 BST 2014 armv6l GNU/Linux\n"
},
"intervals": [],
"end": {
},
"error": "interrupt - the server has terminated"
}
</RECV_OUTPUT>
<SEND_OUTPUT>
{
"start": {
"connected": [{
"socket": 16,
"local_host": "192.168.129.94",
"local_port": 42630,
"remote_host": "192.168.129.81",
"remote_port": 5665
}],
"version": "iperf 3.0.9",
"system_info": "Linux osboxes 3.10.0-123.13.1.el7.x86_64 #1
SMP Tue Dec 9 23:06:09 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux\n",
"timestamp": {
"time": "Fri, 19 Dec 2014 20:14:15 GMT",
"timesecs": 1419020055
},
"connecting_to": {
"host": "192.168.129.81",
"port": 5665
},
"cookie": "osboxes.1419020055.082847.65ec9a0b4c",
"test_start": {
"protocol": "UDP",
"num_streams": 1,
"blksize": 8192,
"omit": 0,
"duration": 2,
"bytes": 0,
"blocks": 0,
"reverse": 0
}
},
"intervals": [{
"streams": [{
"socket": 16,
"start": 0,
"end": 1.00102,
"seconds": 1.00102,
"bytes": 565248,
"bits_per_second": 4.51736e+06,
"packets": 69,
"omitted": false
}],
"sum": {
"start": 0,
"end": 1.00102,
"seconds": 1.00102,
"bytes": 565248,
"bits_per_second": 4.51736e+06,
"packets": 69,
"omitted": false
}
}, {
"streams": [{
"socket": 16,
"start": 1.00102,
"end": 2.00032,
"seconds": 0.999296,
"bytes": 630784,
"bits_per_second": 5.04983e+06,
"packets": 77,
"omitted": false
}],
"sum": {
"start": 1.00102,
"end": 2.00032,
"seconds": 0.999296,
"bytes": 630784,
"bits_per_second": 5.04983e+06,
"packets": 77,
"omitted": false
}
}],
"end": {
"streams": [{
"udp": {
"socket": 16,
"start": 0,
"end": 2.00032,
"seconds": 2.00032,
"bytes": 1196032,
"bits_per_second": 4.78337e+06,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 146,
"lost_percent": 0
}
}],
"sum": {
"start": 0,
"end": 2.00032,
"seconds": 2.00032,
"bytes": 1196032,
"bits_per_second": 4.78337e+06,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 146,
"lost_percent": 0
},
"cpu_utilization_percent": {
"host_total": 4.32541,
"host_user": 0.200171,
"host_system": 4.30456,
"remote_total": 1.61363,
"remote_user": 0,
"remote_system": 1.61363
}
},
"server_output_json": {
"start": {
"connected": [{
"socket": 16,
"local_host": "192.168.129.81",
"local_port": 5665,
"remote_host": "192.168.129.94",
"remote_port": 42630
}],
"version": "iperf 3.0.10",
"system_info": "Linux raspberrypi 3.12.28+ #709
PREEMPT Mon Sep 8 15:28:00 BST 2014 armv6l GNU/Linux\n",
"timestamp": {
"time": "Fri, 19 Dec 2014 20:14:14 GMT",
"timesecs": 1419020054
},
"accepted_connection": {
"host": "192.168.129.94",
"port": 41644
},
"cookie":
"osboxes.1419020055.082847.65ec9a0b4c",
"test_start": {
"protocol": "UDP",
"num_streams": 1,
"blksize": 8192,
"omit": 0,
"duration": 2,
"bytes": 0,
"blocks": 0,
"reverse": 0
}
},
"intervals": [{
"streams": [{
"socket": 16,
"start": 0,
"end": 1.00023,
"seconds": 1.00023,
"bytes": 565248,
"bits_per_second":
4520940,
"jitter_ms": 4.57404,
"lost_packets": 0,
"packets": 69,
"lost_percent": 0,
"omitted": false
}],
"sum": {
"start": 0,
"end": 1.00023,
"seconds": 1.00023,
"bytes": 565248,
"bits_per_second": 4520940,
"jitter_ms": 4.57404,
"lost_packets": 0,
"packets": 69,
"lost_percent": 0,
"omitted": false
}
}, {
"streams": [{
"socket": 16,
"start": 1.00023,
"end": 2.00021,
"seconds": 0.999982,
"bytes": 630784,
"bits_per_second":
5046360,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 77,
"lost_percent": 0,
"omitted": false
}],
"sum": {
"start": 1.00023,
"end": 2.00021,
"seconds": 0.999982,
"bytes": 630784,
"bits_per_second": 5046360,
"jitter_ms": 5.31038,
"lost_packets": 0,
"packets": 77,
"lost_percent": 0,
"omitted": false
}
}],
"end": {
}
}
}
</SEND_OUTPUT>

On 19 Dec 2014, at 19:06, Aaron Brown <> wrote:

Hey Gaetano,

Could you send the full output you’re seeing?

Cheers,

Aaron

On Dec 19, 2014, at 12:01 PM, Gaetano Vocca <> wrote:

Hi Aaron,

thank you for the patch, I applied it on iperf3.c in the folder bwlib.

Now it works well, I am able to get the test done in both cases (sender and receiver)!

Unfortunately one more problem pops up in the JSON output which is the one I am using for parsing. The structure is not correct due to one exceeding closing curly brace. I don’t know if this is the right mailing list to highlight this issue, thank you for any suggestion you may have.

Thanks again

Gaetano

Da: Aaron Brown [mailto:]
Inviato: venerdì 19 dicembre 2014 16:06
A: Gaetano Vocca
Cc: Alan Whinery;
Oggetto: Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi

Hey Gaetano,

Could you try applying this patch to bwctl and running it? It uses the iperf3 binary instead of the library:

http://ndb1.internet2.edu/~aaron/bwctl_change_iperf3_exec.patch

You should be able to apply it by doing “patch -i /path/to/bwctl_change_iperf3_exec.patch -p1” while in the bwctl directory.

Cheers,

Aaron

On Dec 19, 2014, at 9:53 AM, Gaetano Vocca <> wrote:

You are right, bwctld needed a restart.

By running

bwctl -T iperf -u -c <receiver ip>

I get the expected results.

To recap, on my Raspberry Pi running latest Raspbian:

1)      bwctl –T iperf3 -u –c <ip> leads to a segmentation fault

2)      bwctl –T iperf –u -c <ip> works fine

3)      bwctl –T iperf3 –c <ip> works fine

Thanks

Gaetano

Da: Aaron Brown [mailto:]
Inviato: venerdì 19 dicembre 2014 15:10
A: Gaetano Vocca
Cc: Alan Whinery;
Oggetto: Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi

Hi Geatano,

You’ll need to restart bwctld after you install iperf. It only checks on startup if the various tools are available.

Cheers,

Aaron

On Dec 19, 2014, at 8:37 AM, Gaetano Vocca <> wrote:

Hi Aaron,

you are right, I tried to perform the exact test you suggested but I encountered a problem.

When I issue the command

bwctl -T iperf -y J -u -c <receiver ip>

from the Raspberry I get:

bwctl: Requested tools supporting the requested options not available by both servers. See the '-T' option

bwctl: Available tools that support the requested options: iperf3

I also compiled bwctl by running

./configure --enable-iperf --enable-nuttcp --with-iperf3=/usr/local/bin/iperf3

to explicitly support iperf. So I thought to go by direct call to iperf3.

Please let me know if there is anything else I can do to help investigating.

Gaetano

Da: Aaron Brown [mailto:]
Inviato: venerdì 19 dicembre 2014 13:35
A: Gaetano Vocca
Cc: Alan Whinery;
Oggetto: Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi

Hi Gaetano,

On Dec 19, 2014, at 6:09 AM, Gaetano Vocca <> wrote:

Hi,

thank you, I was able to install iperf on the Raspberry by means pf apt-get.

Coming to Aaron’s question below, I was able to run iperf3

iperf3 -u -c <always the same receiver>

from the Raspberry and it worked, so it seems more a bwctl issue, correct?

I’m trying to narrow it down to bwctl or bwctl when running iperf3, which is why i suggested doing a specifically bwctl/iperf test.

Cheers,

Aaron

Have a nice day

Gaetano

Da:  [mailto:] Per conto di Alan Whinery
Inviato: giovedì 18 dicembre 2014 23:51
A:
Oggetto: Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi

I usually get iperf by doing "apt-get install iperf" on a standard Raspbian install...

On 12/18/2014 12:34 PM, Gaetano Vocca wrote:

Thank you but I am actually looking for Iperf in order to follow Aaron 's suggestion.

Thanks

Il giorno 18/dic/2014, alle ore 23:30, Brian Tierney <> ha scritto:

iperf3 is here: https://github.com/esnet/iperf

On Thu, Dec 18, 2014 at 2:22 PM, Gaetano Vocca <> wrote:

Hi Aaron,
Thank you for your kind answer.
I will surely try but would you please point me to the repository for Iperf?

Thank you
Gaetano

> Il giorno 18/dic/2014, alle ore 22:13, Aaron Brown <> ha scritto:

>
> Hi Geatano,
>
> Could you try running it with iperf or nuttcp instead of iperf3, to see if it’s a bwctl issue or an iperf3 issue.
>
> Cheers,
> Aaron
>
>> On Dec 18, 2014, at 2:11 PM,  wrote:
>>
>> Hello,
>> this is my first post and I would like to say hi to the list members and thank
>> you for all the nice work about the bwctl suite.
>>
>> It is indeed about bwctld that I am writing here because I am trying to run it
>> on a Raspberry Pi with latest Raspbian (Sept 2014).
>> I downloaded the latest (at time) sources (1.5.2) and I went through the whole
>> building flow without any problem. I am able to run tests in almost any way
>> except for the case in which the Raspberry is the "sending" machine (e. g.
>> bwctl -T iperf3 -u -c <receiver ip>) of UDP traffic.
>> In fact, in such case, I find in the logs
>>
>> raspberrypi bwctld[2589]: FILE=endpoint.c, LINE=1455, _BWLEndpointStatus:
>> Child #2590 exited from signal #11
>>
>> This is deterministic, it happens all the times but only when running the test
>> from the Raspberry with -u and -c. I tested the same code on a regular PC and
>> it worked fine.
>>
>> Do you have an idea of what could be the issue?
>>
>> Thank you in advance
>> Gaetano
>

--

Brian Tierney, http://www.es.net/tierney
Energy Sciences Network (ESnet), Berkeley National Lab
http://fasterdata.es.net

Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi, (continued)

List archive

Re: [perfsonar-user] BWCTLD segmantation fault on Raspberry Pi