perfSONAR User Q&A and Other Discussion

[Andi Bergen <>]

Hi,
I am having problems getting the bwctl to run successfully between two
of my install locations.

I followed the steps outlined here:
https://fasterdata.es.net/performance-testing/perfsonar/ps-howto/level-1/
and have this installed on two VMs.

I downloaded the limits file from:
http://stats.es.net/sample_configs/bwctld.limits
(which does include a /16 address for our institution).

What doesn't work is running
bwctl -vv -c 142.104.17.140:4823 -f m -x -a 100
or bwctl -vv -c 142.104.17.140 -f m -x -a 100

However, running this TO any other machine (I found listed here
http://stats.es.net/ServicesDirectory/#search=es) works.

It also works just fine when using the internal IP addresses of these
machines since they are on the same network.

I checked that bwctl is bound on the external interface. And
connections are going through, but then fail (please see attached
/var/log/messages files for both machines).

All UDP and TCP ports are open for machines in the same security
group, as well as all ports I could find in the docs are open
otherwise. The ports mentioned in the log files are opened.

Attached are the /var/log/messages from the VM that is running the
test (client.log) and the machine that functions as the "server"
(server.log).

Did I follow the wrong instructions, or am I missing something
obvious? I am somewhat at a loss for what to check next...

Thank you,
Andi
ps.
Command line output from "client" bwctl -vv -c 142.104.17.140:4823 -f
m -x -a 100
(at a different time than the attached log files, but it is always the
same except the time stamps of course):
bwctl[122582]: FILE=bwctl.c, LINE=2931, Using 10.6.9.19 as the address
for local sender
bwctl[122582]: FILE=bwctl.c, LINE=2931, Using 142.104.17.140:4823 as
the address for remote receiver
bwctl[122582]: FILE=bwctl.c, LINE=2977, Available in-common: iperf nuttcp 
iperf3
bwctl[122582]: FILE=bwctl.c, LINE=3002, Using tool: iperf
bwctl[122582]: FILE=bwctl.c, LINE=3117, Requested Time: 1409679438.153914
bwctl[122582]: FILE=bwctl.c, LINE=3119, Latest Acceptable Time:
1409680038.153914
bwctl[122582]: FILE=bwctl.c, LINE=3292,
Reservation(142.104.17.140:4823): 1409679448.061957
bwctl[122582]: FILE=bwctl.c, LINE=3156, Server '142.104.17.140:4823'
accepted test request at time 1409679448.061957
bwctl[122582]: FILE=bwctl.c, LINE=3292, Reservation(localhost):
1409679448.061957
bwctl[122582]: FILE=bwctl.c, LINE=3185, Client 'localhost' accepted
test request at time 1409679448.061957
bwctl[122582]: FILE=bwctl.c, LINE=3359, BWLStartSessions: Failed
Sep  2 13:01:01 centos bwctld[122131]: FILE=sapi.c, LINE=353, Connection to (10.6.9.20:4823) from (142.104.17.139:51640)
Sep  2 13:01:01 centos bwctld[122131]: FILE=sapi.c, LINE=510, ControlSession([10.6.9.20]:4823) accepted from userid(nil):([142.104.17.139]:51640)
Sep  2 13:01:04 centos bwctld[1300]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1409677264, Fuzz: 5.539062, Reservation Start: 1409677271, Reservation End: 1409677293, Test Start Time: 1409677276
Sep  2 13:01:04 centos bwctld[1300]: FILE=bwctld.c, LINE=442, Reservation Status: time=1409677264 action=new sender=10.6.9.19 receiver=142.104.17.140 tool=iperf res_start=1409677271 res_end=1409677292 test_start=1409677276
Sep  2 13:01:04 centos bwctld[1300]: FILE=bwctld.c, LINE=822, Time Slots
Sep  2 13:01:04 centos bwctld[1300]: FILE=bwctld.c, LINE=827, Time Slot 1: 1409677271 to 1409677293: 1 reservations
Sep  2 13:01:05 centos bwctld[1300]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1409677265, Fuzz: 5.539062, Reservation Start: 1409677272, Reservation End: 1409677294, Test Start Time: 1409677277
Sep  2 13:01:05 centos bwctld[1300]: FILE=bwctld.c, LINE=442, Reservation Status: time=1409677265 action=reschedule sender=10.6.9.19 receiver=142.104.17.140 tool=iperf res_start=1409677272 res_end=1409677293 test_start=1409677277
Sep  2 13:01:05 centos bwctld[1300]: FILE=bwctld.c, LINE=822, Time Slots
Sep  2 13:01:05 centos bwctld[1300]: FILE=bwctld.c, LINE=827, Time Slot 1: 1409677272 to 1409677294: 1 reservations
Sep  2 13:01:05 centos bwctld[122131]: FILE=endpoint.c, LINE=319, bind([142.104.17.140]:6019): Cannot assign requested address
Sep  2 13:01:05 centos bwctld[122131]: FILE=protocol.c, LINE=2056, _BWLWriteStopSession called in wrong state.
Sep  2 13:01:05 centos bwctld[1300]: FILE=bwctld.c, LINE=442, Reservation Status: time=1409677265 action=remove sender=10.6.9.19 receiver=142.104.17.140 tool=iperf res_start=1409677272 res_end=1409677293 test_start=1409677277
Sep  2 13:01:05 centos bwctld[122131]: FILE=bwctld.c, LINE=1471, Control session terminated abnormally...

(bwctl -vv -c 142.104.17.140:4823 -f m -x -a 100)


Sep  2 13:01:03 centos bwctld[121881]: FILE=sapi.c, LINE=353, Connection to (10.6.9.19:4823) from (10.6.9.19:58285)
Sep  2 13:01:03 centos bwctld[121881]: FILE=sapi.c, LINE=510, ControlSession([10.6.9.19]:4823) accepted from userid(nil):([10.6.9.19]:58285)
Sep  2 13:01:03 centos bwctl[121869]: FILE=bwctl.c, LINE=2931, Using 10.6.9.19 as the address for local sender
Sep  2 13:01:03 centos bwctl[121869]: FILE=bwctl.c, LINE=2931, Using 142.104.17.140:4823 as the address for remote receiver
Sep  2 13:01:03 centos bwctl[121869]: FILE=bwctl.c, LINE=2977, Available in-common: iperf nuttcp iperf3 
Sep  2 13:01:03 centos bwctl[121869]: FILE=bwctl.c, LINE=3002, Using tool: iperf
Sep  2 13:01:03 centos bwctl[121869]: FILE=bwctl.c, LINE=3117, Requested Time: 1409677263.505184
Sep  2 13:01:03 centos bwctl[121869]: FILE=bwctl.c, LINE=3119, Latest Acceptable Time: 1409677863.505184
Sep  2 13:01:04 centos bwctl[121869]: FILE=bwctl.c, LINE=3292, Reservation(142.104.17.140:4823): 1409677276.765834
Sep  2 13:01:04 centos bwctl[121869]: FILE=bwctl.c, LINE=3156, Server '142.104.17.140:4823' accepted test request at time 1409677276.765834
Sep  2 13:01:05 centos bwctld[1292]: FILE=bwctld.c, LINE=791, Test Reservation Information: Current Time: 1409677265, Fuzz: 5.554688, Reservation Start: 1409677272, Reservation End: 1409677294, Test Start Time: 1409677277
Sep  2 13:01:05 centos bwctld[1292]: FILE=bwctld.c, LINE=442, Reservation Status: time=1409677265 action=new sender=10.6.9.19 receiver=142.104.17.140 tool=iperf res_start=1409677272 res_end=1409677293 test_start=1409677277
Sep  2 13:01:05 centos bwctld[1292]: FILE=bwctld.c, LINE=822, Time Slots
Sep  2 13:01:05 centos bwctld[1292]: FILE=bwctld.c, LINE=827, Time Slot 1: 1409677272 to 1409677294: 1 reservations
Sep  2 13:01:05 centos bwctl[121869]: FILE=bwctl.c, LINE=3292, Reservation(localhost): 1409677277.899032
Sep  2 13:01:05 centos bwctl[121869]: FILE=bwctl.c, LINE=3185, Client 'localhost' accepted test request at time 1409677277.899032
Sep  2 13:01:05 centos bwctl[121869]: FILE=bwctl.c, LINE=3191, Tests accepted at different times re-requesting test with new time
Sep  2 13:01:05 centos bwctl[121869]: FILE=bwctl.c, LINE=3292, Reservation(142.104.17.140:4823): 1409677277.899032
Sep  2 13:01:05 centos bwctl[121869]: FILE=bwctl.c, LINE=3156, Server '142.104.17.140:4823' accepted test request at time 1409677277.899032
Sep  2 13:01:05 centos bwctl[121869]: FILE=bwctl.c, LINE=3359, BWLStartSessions: Failed
Sep  2 13:01:05 centos bwctld[1292]: FILE=bwctld.c, LINE=442, Reservation Status: time=1409677265 action=remove sender=10.6.9.19 receiver=142.104.17.140 tool=iperf res_start=1409677272 res_end=1409677293 test_start=1409677277