perfSONAR User Q&A and Other Discussion

[Elicia Heera <>]

Hi,

While updating to 4.0.2, the nodes on our network that have been upgraded to CentOS7 gave the following error:

reason:         pidfile.py:42:__exit__:OSError: [Errno 13] Permission denied: '/var/run/pscheduler-ticker.pid'
cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn
executable:     /usr/libexec/pscheduler/daemons/ticker
package:        pscheduler-server-1.0.1.1-1.el7.centos
component:      pscheduler-server
pid:            2599
hostname:       perfsonar3.sanren.ac.za
count:          2
abrt_version:   2.1.11
analyzer:       Python
architecture:   x86_64
duphash:        19ac0e6592662b7d3f6aec6306f9126a1f8678c5
event_log:
kernel:         3.10.0-514.26.2.el7.x86_64
last_occurrence: 1512034095
os_release:     CentOS Linux release 7.4.1708 (Core)
pkg_arch:       noarch
pkg_epoch:      0
pkg_fingerprint: 868D DF2C 9D7B 9686
pkg_name:       pscheduler-server
pkg_release:    1.el7.centos
pkg_vendor:     (none)
pkg_version:    1.0.1.1
runlevel:       N 3
time:           Thu 30 Nov 2017 11:26:43 AM SAST
type:           Python
uid:            1000
username:       pscheduler
uuid:           19ac0e6592662b7d3f6aec6306f9126a1f8678c5

When I checked the status of the pscheduler services, the ticker and scheduler had the following errors:

● pscheduler-scheduler.service - pScheduler server - scheduler

   Loaded: loaded (/usr/lib/systemd/system/pscheduler-scheduler.service; enabled; vendor preset: disabled)

   Active: active (running) since Thu 2017-11-30 11:44:28 SAST; 3h 16min ago

 Main PID: 2839 (scheduler)

   CGroup: /system.slice/pscheduler-scheduler.service

           └─2839 /usr/bin/python /usr/libexec/pscheduler/daemons/scheduler --daemon --pid-file /var/run/pscheduler-schedul...

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Program threw an exception after 0...6703

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Exception: DatabaseError: server c...edly

                                                                 This probably means the server terminated abnormally

                                                                 before or while processing the request....

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Waiting 0.5 seconds before restarting

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Restarting: ['/usr/libexec/pschedu...sn']

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: scheduler INFO     Started

Nov 30 11:45:25 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Program threw an exception after 0...7994

Nov 30 11:45:25 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Exception: DatabaseError: server c...edly

                                                                 This probably means the server terminated abnormally

                                                                 before or while processing the request....

Nov 30 11:45:25 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Waiting 0.75 seconds before restarting

Nov 30 11:45:25 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Restarting: ['/usr/libexec/pschedu...sn']

Nov 30 11:45:25 perfsonar5.sanren.ac.za scheduler[2839]: scheduler INFO     Started

● pscheduler-ticker.service - pScheduler server - ticker

   Loaded: loaded (/usr/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: disabled)

   Active: active (running) since Thu 2017-11-30 11:44:28 SAST; 3h 16min ago

 Main PID: 2836 (ticker)

   CGroup: /system.slice/pscheduler-ticker.service

           └─2836 /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid...

Nov 30 11:44:28 perfsonar5.sanren.ac.za systemd[1]: Starting pScheduler server - ticker...

Nov 30 11:44:28 perfsonar5.sanren.ac.za systemd[1]: Started pScheduler server - ticker.

Nov 30 11:44:43 perfsonar5.sanren.ac.za ticker[2836]: ticker WARNING  Queue maintainer got exception server closed the...tedly

                                                              This probably means the server terminated abnormally

                                                              before or while processing the request....

Nov 30 11:44:46 perfsonar5.sanren.ac.za ticker[2836]: safe_run/ticker ERROR    Program threw an exception after 0:00:15.533487

Nov 30 11:44:46 perfsonar5.sanren.ac.za ticker[2836]: safe_run/ticker ERROR    Exception: OperationalError: terminatin...mmand

                                                      server closed the connection unexpectedly

                                                              This probably means the server terminated abnormally...

Nov 30 11:44:46 perfsonar5.sanren.ac.za ticker[2836]: safe_run/ticker ERROR    Waiting 0.25 seconds before restarting

Nov 30 11:44:46 perfsonar5.sanren.ac.za ticker[2836]: safe_run/ticker ERROR    Restarting: ['/usr/libexec/pscheduler/d...dsn']

When I stopped and started these two services, the errors went away but came back as soon as the node was restarted. 

I don't think this has anything to do with version 4.0.2. Any ideas on what could be causing this and how it could be resolved?

Kind regards

Elicia Heera

SANReN Engineer