perfSONAR User Q&A and Other Discussion

["Warnke, Eric E" <>]

We are rolling out a basic perfSonar net and have run into a snag.  Things
were working as expected for a few days and then Saturday morning at 1am
regular testing stopped.  I double checked everything and restarted it on
Sunday and Monday at 1am everything stopped.  I tried again yesterday to
confirm and restart everything and like clockwork it stopped at 1am this
morning.

Hardware is shuttle DS437, 30GB mSATA SSD disk, 2GB ram, and dual 1g
ethernet.  Fresh install of the toolkit last week via the netinstall.
They are on the low-end of the spectrum, but I can find no evidence that
the hardware is being stressed.

The only change between it working and not working were the following
packages installed on Friday.

perl-perfSONAR_PS-Toolkit-SystemEnvironment-3.4.2-8.pSPS.noarch Fri 06 Mar
2015 03:37:56 AM EST
perl-perfSONAR_PS-Toolkit-3.4.2-8.pSPS.noarch Fri 06 Mar 2015 03:37:55 AM
EST


Another oddity that I'm finding is that while the service_watcher is being
run on a regular basis, it doesn't seem to be doing anything.  When run
manually things return to normal.

Log for service watcher seems to indicate it is restarting these services,
but it would seem that it's failing in a way that is not reported since
the error log is empty.

2015/03/05 01:05:01 (6532) INFO> service_watcher:105 main:: - Regular
restart for OWAMP.
2015/03/06 01:05:01 (14909) INFO> service_watcher:105 main:: - Regular
restart for OWAMP.
2015/03/07 01:05:01 (23858) INFO> service_watcher:105 main:: - Regular
restart for OWAMP.
2015/03/07 01:05:06 (23858) INFO> service_watcher:105 main:: - Regular
restart for perfSONAR-PS Regular Testing.
2015/03/08 01:05:01 (28284) INFO> service_watcher:105 main:: - Regular
restart for OWAMP.
2015/03/08 01:05:06 (28284) INFO> service_watcher:105 main:: - Regular
restart for perfSONAR-PS Regular Testing.
2015/03/09 01:05:01 (5198) INFO> service_watcher:105 main:: - Regular
restart for OWAMP.
2015/03/09 01:05:06 (5198) INFO> service_watcher:105 main:: - Regular
restart for perfSONAR-PS Regular Testing.
2015/03/10 01:05:01 (10371) INFO> service_watcher:105 main:: - Regular
restart for OWAMP.
2015/03/10 01:05:06 (10371) INFO> service_watcher:105 main:: - Regular
restart for perfSONAR-PS Regular Testing.


Any ideas?  I have turned off automatic updates for the time being to see
if that is the root cause of if it's a problem with the service_watcher
script.

Cheers,
Eric Warnke
Research IT
SUNY at Albany