perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Kathy Benninger writes:

A second run of ps-migrate-restore.sh restored the archived data, which is now being displayed. Yay!

Unfortunately, some of the pscheduler services seem to need a manual restart after reboot to run cleanly:
    systemctl -l status pscheduler-scheduler
    systemctl -l status pscheduler-archiver
    systemctl -l status pscheduler-ticker
all show errors (copied below) after a reboot. pscheduler-runner starts and runs cleanly without intervention.

The errors are expected behavior if the database server is bounced or not ready when the pScheduler services are started.  The latter is a possibility right after a reboot.  The former happens during certain package upgrades, which will cause the services to lose their connections and restart.  (I have a to-do item to make those exceptions a little less dramatic-looking in the logs.  You’re not the first to notice this.)  The log entries point to one of those or the other.

If you only get one restart, it’s nothing to worry about.  If you’re seeing them repeatedly, there’s a problem, and I can give you a hand troubleshooting it.  Drop me a line off-list and we’ll have a look at it.

One thing to know about restarts is that there’s a backoff time between them to prevent the system from hammering itself if the restarts are because of problems.  The initial backoff time is very short (< 1 second) and increases with each restart until it reaches a minute.  If the service runs longer than the last backoff time, the length is reset to the minimum.  Ergo, if for some reason the scheduler was continuing to die, it may have reached the point where it was sleeping 60 seconds between attempts.

Thanks for any ideas on what to do. I haven't seen this problem on any of the four machines that I've previously upgraded so I'm a bit stumped. (I manage the machine remotely and, while I have a great contact there to help with hands-on, I'm trying to debug as much as I can before I need to pull him back into the fray.)

None of this should require remote hands.  Is there any significant difference between the problem system and the other four (configuration, hardware age, etc.)? 

--Mark