perfSONAR User Q&A and Other Discussion

[Elicia Heera <>]

Hi Mark,

We have been these errors on a few nodes:

reason:         iperf killed by SIGSEGV

cmdline:        iperf -B 192.33.10.254 -s -f b -P 1 -m -p 5272 -t 30

executable:     /usr/bin/iperf

package:        iperf-2.0.10-1.el7

component:      iperf

pid:            31541

pwd:            /

Is this related to the pid error or is this something else?

Kind regards

Elicia Heera

SANReN 

On Thu, Dec 21, 2017 at 9:39 PM, Doug Wussler <> wrote:

We are seeing something similar since installing 4.02:

 

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1513718919

abrt-cli list --since 1513718919

id 9787b881b7a8f9a6aed2a534820f20df514be703

reason:         pidfile.py:48:__exit__:OSError: [Errno 13] Permission denied: '/var/run/pscheduler-archiver.pid'

time:           Wed 20 Dec 2017 08:15:06 AM EST

cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/archiver --daemon --pid-file /var/run/pscheduler-archiver.pid --dsn @/etc/pscheduler/database/database-dsn

package:        pscheduler-server-1.0.2-1.el7.centos

uid:            1000 (pscheduler)

count:          4

Directory:      /var/spool/abrt/Python-2017-12-20-08:15:06-1928

 

….. and in /var/log/httpd/error_log

 

[Tue Dec 19 16:30:04.774618 2017] [:error] [pid 5425] cassandra_db [INFO] Checking/creating column families

[Tue Dec 19 16:30:04.774671 2017] [:error] [pid 5425] cassandra_db [INFO] Checking/creating column families

[Tue Dec 19 16:30:04.777484 2017] [:error] [pid 5425] cassandra_db [INFO] Schema check done

[Tue Dec 19 16:30:04.777524 2017] [:error] [pid 5425] cassandra_db [INFO] Schema check done

[Tue Dec 19 16:30:04.777578 2017] [:error] [pid 5425] cassandra_db [DEBUG] Opening ConnectionPool

[Tue Dec 19 16:30:04.777605 2017] [:error] [pid 5425] cassandra_db [DEBUG] Opening ConnectionPool

[Tue Dec 19 16:30:04.785895 2017] [:error] [pid 5425] cassandra_db [INFO] Connected to ['localhost:9160']

[Tue Dec 19 16:30:04.785948 2017] [:error] [pid 5425] cassandra_db [INFO] Connected to ['localhost:9160']

[Wed Dec 20 08:15:04.044399 2017] [mpm_prefork:notice] [pid 5418] AH00170: caught SIGWINCH, shutting down gracefully

Exception TypeError: "'NoneType' object is not callable" in <bound method Log.__del__ of <pscheduler.log.Log instance at 0x7f0e90e1c638>> ignored

 

 

Doug Wussler

Florida State University 

 

From: <> on behalf of Roderick Mooi <>
Date: Thursday, December 21, 2017 at 7:09 AM
To: Mark Feit <>
Cc: Elicia Heera <>, "" <>, Andrew Lake <>
Subject: Fwd: Re: [perfsonar-user] pscheduler server error pemission denied

 

Hi Mark

This has appeared on 3 of our nodes over the past two days (clean install of perfSONAR v4.0.2-1.el7.centos with auto-updates enabled).

# abrt-cli list --since 1513167682

id dc387fc33182e035e81b9c4c7aed40ee138d31db
reason:         pidfile.py:42:__exit__:OSError: [Errno 13] Permission denied: '/var/run/pscheduler-ticker.pid'
time:           Thu 30 Nov 2017 11:26:50 SAST
cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn
package:        pscheduler-server-1.0.1.1-1.el7.centos
uid:            1000 (pscheduler)
count:          26
Directory:      /var/spool/abrt/Python-2017-11-30-11:26:50-2203

id 37a88a38173c0ccf3c51e6b81525e082ce7f4042
reason:         python2.7 killed by SIGSEGV
time:           Thu 21 Dec 2017 05:38:55 SAST
cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/runner --daemon --pid-file /var/run/pscheduler-runner.pid --dsn @/etc/pscheduler/database/database-dsn
package:        pscheduler-server-1.0.2-1.el7.centos
uid:            1000 (pscheduler)
count:          1
Directory:      /var/spool/abrt/ccpp-2017-12-21-05:38:55-2069

id aa9cbd7244dee91f00ddb0e89fa175998270658e
reason:         run:320:<module>:ZeroDivisionError: float division by zero
time:           Tue 12 Dec 2017 13:41:56 SAST
cmdline:        /usr/bin/python /usr/libexec/pscheduler/commands/../classes/tool/bwctlping/run
package:        pscheduler-tool-bwctlping-1.0.2-1.el7.centos
uid:            1000 (pscheduler)
count:          10
Directory:      /var/spool/abrt/Python-2017-12-12-13:41:56-33353


# ls -al /var/run
lrwxrwxrwx. 1 root root 6 Aug 28 13:40 /var/run -> ../run

Thanks for your assistance!

Regards,

Roderick

On 2017/12/12 10:19 AM, Elicia Heera wrote:

Hi Mark,

 

Apologies for the delay in getting back to you.

 

If I see the error again, I will send you the output you requested. If I check the status of ticker and scheduler, it shows the error dated with when it occurred and does seem to be reoccurring. 

 

Regarding the migration process, I performed a backup, clean install and restore as per the migration instructions (http://docs.perfsonar.net/install_migrate_centos7.html). At the moment we have only migrated 5 nodes out of 18 using this process and I received this error when updating from 4.0.1 to 4.0.2 on the nodes running CentOS 7. I did a manual update of each node and as it was installing, I received the errors (we have automated emails for errors). If errors occur during a manual update, will the update be installed correctly? 

 

Thank you for the information and help. 

 

Kind regards

Elicia

 

On Fri, Dec 1, 2017 at 6:26 PM, Mark Feit <> wrote:

Elicia Heera writes:

 

While updating to 4.0.2, the nodes on our network that have been upgraded to CentOS7 gave the following error:

 

reason:         pidfile.py:42:__exit__:OSError: [Errno 13] Permission denied: '/var/run/pscheduler-ticker.pid'
cmdline:        /usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon --pid-file /var/run/pscheduler-ticker.pid --dsn @/etc/pscheduler/database/database-dsn
executable:     /usr/libexec/pscheduler/daemons/ticker
package:        pscheduler-server-1.0.1.1-1.el7.centos
component:      pscheduler-server

 

The scripts that start the daemons build that file separately with the proper ownership and atomically move it into place, so unless there’s something weird going on like a umask of 2xx, it should be writable.  If you see this again, please send me the output of “ls -al /var/run” and I’ll have a look at it.

 

Also, could you clarify for me whether the migration you did was an in-place upgrade or a backup, clean install and restore per the migration instructions (http://docs.perfsonar.net/install_migrate_centos7.html)?

 

 

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Program threw an exception after 0...6703

Nov 30 11:45:24 perfsonar5.sanren.ac.za scheduler[2839]: safe_run/scheduler ERROR    Exception: DatabaseError: server c...edly

                                                                 This probably means the server terminated abnormally

                                                                 before or while processing the request....

 

A few of those for each daemon are normal surrounding an upgrade, as a couple of the packages force PostgreSQL to be bounced.  The pScheduler daemons are built to be self-restarting if anything goes wrong and see losing the connection to the database as wrong.  I’ve debated having this happen quietly for that case but decided to err on the side of leaving the warning in there in case the database dies for other reasons.

 

--Mark