perfSONAR User Q&A and Other Discussion

[Elicia Heera <>]

Hi Everyone,

I need some help regarding the pScheduler server and some of the errors I am getting and how to resolve them. I recently did a fresh install of perfSONAR on a FIT-PC 3 device. When this device was deployed to site the pScheduler process has been giving numerous errors including Internal error on on local pScheduler server when I run pscheduler task idle --duration PT2S.

Under the pscheduler log:

Oct 16 13:33:38 localhost journal: safe_run/scheduler ERROR    Restarting

Oct 16 13:33:38 localhost journal: safe_run/scheduler ERROR    Program threw an exception after 0:00:00.001245

Oct 16 13:33:38 localhost journal: safe_run/scheduler ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1"$

Oct 16 13:33:38 localhost journal: safe_run/scheduler ERROR    Waiting 19.75 seconds before restarting

Oct 16 13:33:38 localhost journal: safe_run/runner ERROR    Restarting

Oct 16 13:33:38 localhost journal: safe_run/runner ERROR    Program threw an exception after 0:00:00.001344

Oct 16 13:33:38 localhost journal: safe_run/runner ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" an$

Oct 16 13:33:38 localhost journal: safe_run/runner ERROR    Waiting 19.75 seconds before restarting

Oct 16 13:33:38 localhost journal: safe_run/archiver ERROR    Restarting

Oct 16 13:33:38 localhost journal: safe_run/archiver ERROR    Program threw an exception after 0:00:00.001308

Oct 16 13:33:38 localhost journal: safe_run/archiver ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" $

Oct 16 13:33:38 localhost journal: safe_run/archiver ERROR    Waiting 19.75 seconds before restarting

Oct 16 13:33:39 localhost journal: safe_run/ticker ERROR    Restarting

Oct 16 13:33:39 localhost journal: safe_run/ticker ERROR    Program threw an exception after 0:00:00.005828

Oct 16 13:33:39 localhost journal: safe_run/ticker ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" an$

Oct 16 13:33:39 localhost journal: safe_run/ticker ERROR    Waiting 19.75 seconds before restarting

Oct 16 13:33:58 localhost journal: safe_run/scheduler ERROR    Restarting

Oct 16 13:33:58 localhost journal: safe_run/scheduler ERROR    Program threw an exception after 0:00:00.001213

Oct 16 13:33:58 localhost journal: safe_run/scheduler ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1"$

Oct 16 13:33:58 localhost journal: safe_run/scheduler ERROR    Waiting 20.0 seconds before restarting

Oct 16 13:33:58 localhost journal: safe_run/runner ERROR    Restarting

Oct 16 13:33:58 localhost journal: safe_run/runner ERROR    Program threw an exception after 0:00:00.001320

Oct 16 13:33:58 localhost journal: safe_run/runner ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" an$

Oct 16 13:33:58 localhost journal: safe_run/runner ERROR    Waiting 20.0 seconds before restarting

Oct 16 13:33:58 localhost journal: safe_run/archiver ERROR    Restarting

Oct 16 13:33:58 localhost journal: safe_run/archiver ERROR    Program threw an exception after 0:00:00.001256

Oct 16 13:33:58 localhost journal: safe_run/archiver ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" $

Oct 16 13:33:58 localhost journal: safe_run/archiver ERROR    Waiting 20.0 seconds before restarting

Oct 16 13:33:59 localhost journal: safe_run/ticker ERROR    Restarting

Oct 16 13:33:59 localhost journal: safe_run/ticker ERROR    Program threw an exception after 0:00:00.005256

Oct 16 13:33:59 localhost journal: safe_run/ticker ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" an$

Oct 16 13:33:59 localhost journal: safe_run/ticker ERROR    Waiting 20.0 seconds before restarting

I get these errors when checking the status of any of the pscheduler services. All show active (running):

pscheduler-runner.service - pScheduler server - runner

   Loaded: loaded (/usr/lib/systemd/system/pscheduler-runner.service; enabled; vendor preset: disabled)

   Active: active (running) since Mon 2017-10-16 13:20:41 SAST; 16min ago

 Main PID: 1254 (runner)

   CGroup: /system.slice/pscheduler-runner.service

           └─1254 /usr/bin/python /usr/libexec/pscheduler/daemons/runner --daemon --pid-file /var/run/pscheduler-runner.pid --dsn @/etc/pscheduler/database/database-dsn

Oct 16 13:36:02 localhost.localdomain runner[1254]: safe_run/runner ERROR    Exception: OperationalError: could not connect to server: Connection refused

                                                            Is the server running on host "127.0.0.1" and accepting

                                                            TCP/IP connections on port 5432?...

Oct 16 13:36:02 localhost.localdomain runner[1254]: safe_run/runner ERROR    Waiting 21.5 seconds before restarting

Oct 16 13:36:23 localhost.localdomain runner[1254]: safe_run/runner ERROR    Restarting

Oct 16 13:36:23 localhost.localdomain runner[1254]: safe_run/runner ERROR    Program threw an exception after 0:00:00.001386

Oct 16 13:36:23 localhost.localdomain runner[1254]: safe_run/runner ERROR    Exception: OperationalError: could not connect to server: Connection refused

                                                            Is the server running on host "127.0.0.1" and accepting

                                                            TCP/IP connections on port 5432?...

Oct 16 13:36:23 localhost.localdomain runner[1254]: safe_run/runner ERROR    Waiting 21.75 seconds before restarting

Oct 16 13:36:45 localhost.localdomain runner[1254]: safe_run/runner ERROR    Restarting

Oct 16 13:36:45 localhost.localdomain runner[1254]: safe_run/runner ERROR    Program threw an exception after 0:00:00.000983

Oct 16 13:36:45 localhost.localdomain runner[1254]: safe_run/runner ERROR    Exception: OperationalError: could not connect to server: Connection refused

                                                            Is the server running on host "127.0.0.1" and accepting

                                                            TCP/IP connections on port 5432?...

Oct 16 13:36:45 localhost.localdomain runner[1254]: safe_run/runner ERROR    Waiting 22.0 seconds before restarting

I have tried reinstalling the pscheduler server and restarting all the processes with no luck.

Any help would be appreciated.

Sincerly

Elicia Heera

Network Engineer

​