perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

On 11/12/2018 12:54, Mark Feit wrote:
> Brian Candler writes:
>
>      After a "yum upgrade" of a 4.1 node which was a few months old (I think
>      it went for 4.1.2 to 4.1.4), I got the following error after sudo to 
> root:
>      ...
>      reason:         pidfile.py:48:__exit__:OSError: [Errno 13] Permission
>      denied: '/run/pscheduler-ticker.pid'
>      ...
>      uid:            1000 (pscheduler)
>      ...
>      # ls -l /run/pscheduler-ticker.pid
>      -rw-r--r-- 1 pscheduler pscheduler 4 Dec 10 21:45 
> /run/pscheduler-ticker.pid
>      
>      # ls -ln /run/pscheduler-ticker.pid
>      -rw-r--r-- 1 1000 1000 4 Dec 10 21:45 /run/pscheduler-ticker.pid
>      
>      # ls -ld /run
>      drwxr-xr-x 39 root root 1400 Dec 11 08:12 /run
>   
> All of that looks correct.  The PID files are created and chowned by the systemd service file (see /usr/lib/systemd/system/pscheduler-ticker.service) as root prior to the daemon starting.  The last change to anything related to starting programs was in August's 4.1(.0) release.  4.1.3 and 4.1.4 were almost non-events for pScheduler, so this shouldn't be anything related to the upgrade.
>
> Are you seeing this just for the ticker or for the other pScheduler services 
> as well?

I only saw that one message. After another sudo no more such warnings 
are seen, and "abrt-cli list --since 1544476167" doesn't show any more 
messages.


>    Is the system a stock toolkit or have there been site-specifc changes made 
> to the system?

Stock toolkit installed from ISO (although running in a VM - it's just 
for demo purposes)


>    Is there a ticker running?

Yes, although I'm afraid I have restarted the VM since originally posting.

[root@perfsonar0 sysadm]# ps auxwww | grep ticker
pschedu+  4325  0.0  0.5 349192 20484 ?        Ssl  08:38   0:02 
/usr/bin/python /usr/libexec/pscheduler/daemons/ticker --daemon 
--pid-file /run/pscheduler-ticker.pid --dsn 
@/etc/pscheduler/database/database-dsn
root      9555  0.0  0.0 112712   980 pts/0    R+   21:53   0:00 grep 
--color=auto ticker
[root@perfsonar0 sysadm]# uptime
 21:53:33 up 13:14,  1 user,  load average: 0.02, 0.06, 0.11
>    Anything in /var/log/pscheduler/pscheduler.log other than the complaint 
> about the failure?


I don't see the exception which abrt showed.  I do see several cases of 
what look like SQL exceptions:

Dec 12 08:38:56 perfsonar0 journal: ticker INFO     Started
Dec 12 08:39:18 perfsonar0 journal: ticker WARNING  Queue maintainer got 
exception server closed the connection unexpectedly
Dec 12 08:39:18 perfsonar0 journal: ticker WARNING  #011This probably 
means the server terminated abnormally
Dec 12 08:39:18 perfsonar0 journal: ticker WARNING  #011before or while 
processing the request.
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR Program threw 
an exception after 0:00:34.059668
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR Exception: 
OperationalError: server closed the connection unexpectedly
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR #011This 
probably means the server terminated abnormally
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR #011before or 
while processing the request.
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR Traceback 
(most recent call last):
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 73, in 
safe_run
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR function()
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR File 
"/usr/libexec/pscheduler/daemons/ticker", line 157, in <lambda>
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR 
pscheduler.safe_run(lambda: main_program())
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR File 
"/usr/libexec/pscheduler/daemons/ticker", line 137, in main_program
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR 
cursor.execute("SELECT ticker()")
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR 
OperationalError: server closed the connection unexpectedly
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR #011This 
probably means the server terminated abnormally
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR #011before or 
while processing the request.
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR Waiting 0.25 
seconds before restarting
Dec 12 08:39:31 perfsonar0 journal: safe_run/ticker ERROR Restarting: 
['/usr/libexec/pscheduler/daemons/ticker', '--daemon', '--pid-file', 
'/run/pscheduler-ticker.pid', '--dsn', 
'@/etc/pscheduler/database/database-dsn']
Dec 12 08:39:31 perfsonar0 journal: ticker INFO     Started

There are some variations on this, e.g.

Dec 10 21:41:23 perfsonar0 journal: safe_run/ticker ERROR    SQL 
statement "SELECT tool_boot()"
Dec 10 21:41:23 perfsonar0 journal: safe_run/ticker ERROR PL/pgSQL 
function warm_boot() line 5 at PERFORM
Dec 10 21:41:23 perfsonar0 journal: safe_run/ticker ERROR    SQL 
statement "SELECT warm_boot()"
Dec 10 21:41:23 perfsonar0 journal: safe_run/ticker ERROR PL/pgSQL 
function cold_boot() line 3 at PERFORM
Dec 10 21:41:23 perfsonar0 journal: safe_run/ticker ERROR server closed 
the connection unexpectedly

Given that it restarts and recovers, these don't worry me - maybe just 
to do with startup sequencing.

Don't worry - if this abrt report doesn't have any value, and given that 
it's not recurring, then I don't think it's worth further 
investigation.  To be honest I had never come across abrt before :-)

Cheers,

Brian.