perfSONAR User Q&A and Other Discussion

[Jason Zurawski <>]

Hey Trey;

Apologies, I fat fingered the instructions - try a lowercase ’s’ for that 
command to TACC (or just view the -help output from BWCTL to see the command 
line options).  

With regards to the error, perhaps try using ‘-T iperf’ or ‘-T nuttcp’ to see 
if that has any change (e.g. changing the tools used).  The default testing 
tool is now iperf3, but seeing what the others say would be good.  As I noted 
in the previous mail, there are still some funny behaviors the developers are 
sorting out with iperf3 and BWCTL. 

Thanks;

-jason

> On Jan 20, 2015, at 12:57 PM, Trey Dockendorf 
> <>
>  wrote:
> 
> Jason,
> 
> Thanks for the response.
> 
> For #1 the second command doesn't work
> 
> bwctl -T iperf3 -t 20 -i 1 -f m -x -vv -S ps.tacc.utexas.edu
> 
> Gives me:
> 
> bwctl: Invalid value for TOS. (-S)
> 
> Unsure what is valid value would be.
> 
> When I tried the first command against a remote host I got the following:
> 
> # bwctl -T iperf3 -t 20 -i 1 -f m -x -vv -c ps.tacc.utexas.edu
> Messages being sent to syslog(user,err)
> bwctl[30659]: FILE=bwctl.c, LINE=2961, Using 165.91.55.6 as the address for 
> local sender
> bwctl[30659]: FILE=bwctl.c, LINE=2961, Using ps.tacc.utexas.edu as the 
> address for remote receiver
> bwctl[30659]: FILE=bwctl.c, LINE=3008, Available in-common: iperf nuttcp 
> iperf3
> bwctl[30659]: FILE=bwctl.c, LINE=3055, Using tool: iperf3
> bwctl[30659]: FILE=bwctl.c, LINE=3150, Requested Time: 1421776247.770821
> bwctl[30659]: FILE=bwctl.c, LINE=3152, Latest Acceptable Time: 
> 1421776847.770821
> bwctl[30659]: FILE=bwctl.c, LINE=3332, Reservation(ps.tacc.utexas.edu): 
> 1421776254.700447
> bwctl[30659]: FILE=bwctl.c, LINE=3189, Server 'ps.tacc.utexas.edu' accepted 
> test request at time 1421776254.700447
> bwctl[30659]: FILE=bwctl.c, LINE=3332, Reservation(localhost): 
> 1421776254.700447
> bwctl[30659]: FILE=bwctl.c, LINE=3218, Client 'localhost' accepted test 
> request at time 1421776254.700447
> bwctl[30659]: FILE=bwctl.c, LINE=3450, 28 seconds until test results 
> available
> bwctl[30659]: FILE=protocol.c, LINE=506, BWLReadRequestType: Read 
> interrupted by signal.
> bwctl[30659]: FILE=capi.c, LINE=1166, BWLEndSession: Invalid protocol 
> message received...
> bwctl[30659]: FILE=bwctl.c, LINE=3546, Timed out waiting for results
> 
> This particular endpoint is of interest at this time as another group on 
> our campus has asked me to test against the remote site to investigate 
> recent performance issues from their cluster to the remote site.
> 
> I'll continue testing each host that's results are missing and see if 
> errors are non obvious.
> 
> Thanks,
> - Trey
> 
> =============================
> 
> Trey Dockendorf 
> Systems Analyst I 
> Texas A&M University 
> Academy for Advanced Telecommunications and Learning Technologies 
> Phone: (979)458-2396 
> Email: 
> 
>  
> Jabber: 
> 
> 
> On Tue, Jan 20, 2015 at 10:14 AM, Jason Zurawski 
> <>
>  wrote:
> Hey Trey;
> 
> Your mail is timely, we are trying to re-write some documentation on this.  
> This will be sort of scattered, but may be a enough to get you started:
> 
> 1) Take a look at the hosts you have configured vs. the ones that are 
> succeeding.  See if you can do a test by hand to one of the failures (e.g. 
> on the cmd line: bwctl -T iperf3 -t 20 -i 1 -f m -x -vv -c HOST and bwctl 
> -T iperf3 -t 20 -i 1 -f m -x -vv -S HOST).  If the host is dead, doesn’t 
> support the requested test type, or denies your test in some way it is most 
> likely a problem on the other end.  Maybe send an email to that admin to 
> see what is up.
> 
> 2) If the test from the first part succeeded, its time to look in the 
> database.  In your case, open your favorite browser (preferably one with a 
> JSON prettifier in place) to your esmond database:
> 
> http://psonar-bwctl.brazos.tamu.edu/esmond/perfsonar/archive/?format=json
> 
> Each test should have a corresponding record.  The ones that succeed, and 
> the ones that fail.  Picking one that is failing for you (to 
> ps1-akard-dlls.tx-learn.net - the LEARN Dallas node), we can pull up the 
> failure event type:
> 
> http://psonar-bwctl.brazos.tamu.edu/esmond/perfsonar/archive/7f8d5be4115b4927b7ce6a5dfbfa8e44/failures/base?format=json
> 
> In this case we get a (useful?) error message:
> 
> > Problem parsing output: malformed JSON string, neither array, object, 
> > number, string or atom, at character offset 0 (before "(end of string)") 
> > at 
> > /opt/perfsonar_ps/regular_testing/bin/../lib/perfSONAR_PS/RegularTesting/Parsers/Iperf3.pm
> >  line 54.
> 
> 
> This is a known problem right now between certain versions of BWCTL and 
> iperf3 that the developers are working on.  Another common error:
> 
> > interrupt - the client has terminated
> 
> 
> Typically means there was either a firewall or NTP issue.
> 
> 3) Check your server side logs (/var/log/perfsonar) to see if anything is 
> showing up in the bwctl_owamp log.  Common errors could be permission 
> denied due to limits violations, the lack of being able to get a testing 
> slot (common for busy perfSONAR nodes) or the aforementioned firewall 
> issues.
> 
> In general if your node is working for ‘some’ things, it may be on the 
> healthy side.  Things that you are trying to test against may just not be 
> reachable, or could be in need of an upgrade (another common issue we are 
> seeing - until we can get a majority of the instances to 3.4.x).
> 
> Hope this helps;
> 
> -jason
> 
> On Jan 20, 2015, at 10:54 AM, Trey Dockendorf 
> <>
>  wrote:
> 
> > I have about 14 test members added for throughput tests and only 3 of 
> > them show up in the Throughput/Latency Graphs page.  I am unsure how to 
> > begin debugging this to find the cause of the missing data.
> >
> > The host is http://psonar-bwctl.brazos.tamu.edu/.  I have most of these 
> > sites also added on a separate perfsonar instance that does latency tests 
> > and that host is not missing data.
> >
> > Thanks,
> > - Trey
> >
> > =============================
> >
> > Trey Dockendorf
> > Systems Analyst I
> > Texas A&M University
> > Academy for Advanced Telecommunications and Learning Technologies
> > Phone: (979)458-2396
> > Email: 
> > 
> > Jabber: 
> >