perfSONAR User Q&A and Other Discussion

[Roderick Mooi <>]

Hi Mark

This has appeared on 3 of our nodes over the past two days (clean install of perfSONAR v4.0.2-1.el7.centos with auto-updates enabled).

# abrt-cli list --since 1513167682

id dc387fc33182e035e81b9c4c7aed40ee138d31db
reason:         pidfile.py:42:__exit__:OSError: [Errno 13] Permission denied: '/var/run/pscheduler-ticker.pid'
time:           Thu 30 Nov 2017 11:26:50 SAST
cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn
package:        pscheduler-server-1.0.1.1-1.el7.centos
uid:            1000 (pscheduler)
count:          26
Directory:      /var/spool/abrt/Python-2017-11-30-11:26:50-2203

id 37a88a38173c0ccf3c51e6b81525e082ce7f4042
reason:         python2.7 killed by SIGSEGV
time:           Thu 21 Dec 2017 05:38:55 SAST
cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/runner --daemon --pid-file /var/run/pscheduler-runner.pid --dsn @/etc/pscheduler/database/database-dsn
package:        pscheduler-server-1.0.2-1.el7.centos
uid:            1000 (pscheduler)
count:          1
Directory:      /var/spool/abrt/ccpp-2017-12-21-05:38:55-2069

id aa9cbd7244dee91f00ddb0e89fa175998270658e
reason:         run:320:<module>:ZeroDivisionError: float division by zero
time:           Tue 12 Dec 2017 13:41:56 SAST
cmdline:        /usr/bin/python /usr/libexec/pscheduler/commands/../classes/tool/bwctlping/run
package:        pscheduler-tool-bwctlping-1.0.2-1.el7.centos
uid:            1000 (pscheduler)
count:          10
Directory:      /var/spool/abrt/Python-2017-12-12-13:41:56-33353


# ls -al /var/run
lrwxrwxrwx. 1 root root 6 Aug 28 13:40 /var/run -> ../run

Thanks for your assistance!

Regards,

Roderick

On 2017/12/12 10:19 AM, Elicia Heera wrote:

Hi Mark,

Apologies for the delay in getting back to you.

If I see the error again, I will send you the output you requested. If I check the status of ticker and scheduler, it shows the error dated with when it occurred and does seem to be reoccurring. 

Regarding the migration process, I performed a backup, clean install and restore as per the migration instructions (http://docs.perfsonar.net/install_migrate_centos7.html). At the moment we have only migrated 5 nodes out of 18 using this process and I received this error when updating from 4.0.1 to 4.0.2 on the nodes running CentOS 7. I did a manual update of each node and as it was installing, I received the errors (we have automated emails for errors). If errors occur during a manual update, will the update be installed correctly? 

Thank you for the information and help. 

Kind regards

Elicia

On Fri, Dec 1, 2017 at 6:26 PM, Mark Feit <> wrote:

Elicia Heera writes:

 

While updating to 4.0.2, the nodes on our network that have been upgraded to CentOS7 gave the following error:

 

reason:         pidfile.py:42:__exit__:OSError: [Errno 13] Permission denied: '/var/run/pscheduler-ticker.pid'
cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn
executable:     /usr/libexec/pscheduler/daemons/ticker
package:        pscheduler-server-1.0.1.1-1.el7.centos
component:      pscheduler-server

 

The scripts that start the daemons build that file separately with the proper ownership and atomically move it into place, so unless there’s something weird going on like a umask of 2xx, it should be writable.  If you see this again, please send me the output of “ls -al /var/run” and I’ll have a look at it.

 

Also, could you clarify for me whether the migration you did was an in-place upgrade or a backup, clean install and restore per the migration instructions (http://docs.perfsonar.net/install_migrate_centos7.html)?

 

 

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Program threw an exception after 0...6703

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Exception: DatabaseError: server c...edly

                                                                 This probably means the server terminated abnormally

                                                                 before or while processing the request....

 

A few of those for each daemon are normal surrounding an upgrade, as a couple of the packages force PostgreSQL to be bounced.  The pScheduler daemons are built to be self-restarting if anything goes wrong and see losing the connection to the database as wrong.  I’ve debated having this happen quietly for that case but decided to err on the side of leaving the warning in there in case the database dies for other reasons.

 

--Mark