perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Group,

     I have a mesh of 4 PerfSonar nodes (and 1 collector) in a mesh.  last Thursday morning (9th) at 1:00am, Owamp collection for half of the hosts in the mesh stopped.  I'm still collecting bandwidth data, and traceroute data, just no Owamp.  at around 1:00am on Friday, the other half stopped.

     I can manually run latency testing between hosts using bwctl using either ping or owamp as the tool with no problems.  However, in the maddash interfaces,  if you try to look at the details for the recent tests you get:

"Unable to find any tests in the given time range where....."

I've re-pulled the mesh config on all the testing hosts.  I've restarted the regular testing daemon on all the testing hosts.  I've restarted the local latency services.  I eventually restarted the hosts.  All to no effect.

Since it occurred on two consecutive nights at 1:00am I suspected it was an automatic update that caused the problem.  So I checked the IPtables rules that were borked in the original 3.4x release.  Although they've been all re-written since my original installs, they seem legitimate. 

I did have to make changes to the /etc/httpd/conf.d/apache-toolkit_web_gui.conf to re-enable our Radius authentication for the web interface on the boxes since a recent update had overwritten it.  But besides that, I'm unable to find anything in my own looking that would have caused this problem.

I'm going to need help from someone that knows PS a lot better than I.  I'll be happy to share any log files you need to help things along.  The dashboard is at:  http://ps-dashboard.perfsonar.kanren.net/maddash-webui/ 

Thank you,

Casey Russell

Network Engineer

Kansas Research and Education Network

2029 Becker Drive, Suite 282

Lawrence, KS  66047

(785)856-9820  ext 9809