perfSONAR User Q&A and Other Discussion

[Brian Tierney <>]

On Feb 2, 2011, at 11:49 AM, Carl Hayter wrote:

> On Wed, Feb 02, 2011 at 11:04:49AM -0500, Brian Tierney wrote:
>> 
>> There is an updated 'service_watcher' script that should makes sure it 
>> will stay up.
> 
> Does this have anything remotely to do with my current problem:

Yes, most likely its related. There seems to be a bug in the xmldb package
we are using. As a work-around the 3.2 toolkit install creates these cron 
entries:

0 * * * * root /opt/perfsonar_ps/toolkit/scripts/service_watcher &> 
/var/log/perfsonar/service_watcher_error.log
0 1 * * * root /opt/perfsonar_ps/toolkit/scripts/service_watcher 
--regular_restart &> /var/log/perfsonar/service_watcher_error.log

The 2nd one will reset the database nightly.

Can you verify that you have these cron jobs in your config?



> 
> After a couple of days running, my services stop showing up in the
> ls_cache files (I'm assuming this is the same sort of data that the web
> page is displaying).  In my logs I see these:
> 
> 2011/02/01 19:01:48 (18066) ERROR> XMLDB.pm:248
> perfSONAR_PS::DB::XMLDB::openDB - Error "Error: Db::open:
> DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock in
> /opt/perfsonar_ps/lookup_service/bin/../lib/perfSONAR_PS/DB/XMLDB.pm,
> line 231".
> 
> The only way I've found to get things back to being registered is to
> stop the lookup_service and ls_registration_daemon, delete everything
> but the DB_CONFIG from /var/lib/perfsonar/lookup_service/xmldb directory
> and restart the lookup/registration services.  After a few hours I'm
> back in the cache for a day or two with plenty of:
> 
> 2011/02/01 19:02:14 (18080) ERROR> XMLDB.pm:248
> perfSONAR_PS::DB::XMLDB::openDB 
> - Error "Error: DB_LOCK_NOTGRANTED: Lock not granted in
> /opt/perfsonar_ps/l
> ookup_service/bin/../lib/perfSONAR_PS/DB/XMLDB.pm, line 231".
> 
> I forget which version of PerfSONAR I started with, but I've been
> keeping up to date (hopefully) with yum check-update ...
> 
> And I also installed remotely without using the CD install.  Did a bare
> network install of CentOS 5.5 and then added repositories and packages
> copied from a VirtualBox install of the CD version and the
> anaconda-ks.cfg listing.  Aside from the problems I'm having with the
> registration parts everything seems good on my end.
> 
> I'm currently visible on the web page, the only machines in the CENIC
> group (University of Southern California) at  https://raharu.usc.edu/toolkit
> running lookkup and bandwith services.
> 
> Not sure if I missed some important configuration thing or what keeps my
> registrations from working.
> 
> ____
> Carl Hayter
> ITS - Data Network Operations
> University of Southern California