perfSONAR User Q&A and Other Discussion

[Simon peter green <>]

Hi all,

Currently unable to run tests via pscheduler due to the following error that occurs:

$ pscheduler task throughput --source bwctl-10g-ps.singaren.net.sg --dest ps-londhx1.ja.net -t 20 --omit 10 

Submitting task...

Unable to post task: function api_task_post(unknown, text[], unknown, unknown, integer, unknown, boolean) is not unique

Received the following error when running a reverse test as shown below:

$ pscheduler task throughput --dest 203.30.39.13 --source ps-londhx1.ja.net

Submitting task...

Unable to post task: Error while tasking 203.30.39.13: Unable to post task to 203.30.39.13: function api_task_post(unknown, text[], unknown, unknown, integer, unknown, boolean) is not unique

LINE 1: SELECT * FROM api_task_post('{"test": {"type": "throughput",...

                      ^

HINT:  Could not choose a best candidate function. You might need to add explicit type casts.

It seems there are issues with the database; I’ve checked pscheduler.log and there are indeed database errors:

Mar 10 00:31:49 owamp-ps safe_run/runner ERROR    Exception: DatabaseError: server closed the connection unexpectedly#012#011This probably means the server terminated abnormally#012#011before or while processing the request.#012#012Traceback (most recent call last):#012  

File "/usr/lib/python2.6/site-packages/pscheduler/saferun.py", line 72, in safe_run#012    function()#012  

File "/usr/libexec/pscheduler/daemons/runner", line 933, in <lambda>#012    pscheduler.safe_run(lambda: main_program())#012  

File "/usr/libexec/pscheduler/daemons/runner", line 842, in main_program#012    """, [refresh]);#012DatabaseError: server closed the connection unexpectedly#012#011This probably means the server terminated abnormally#012#011before or while processing the request.

Mar 10 00:31:49 owamp-ps safe_run/scheduler ERROR    Program threw an exception after 5:52:07.161252

Mar 10 00:31:49 owamp-ps safe_run/runner ERROR    Waiting 6.5 seconds before restarting

Mar 10 00:31:49 owamp-ps ticker WARNING  Queue maintainer got exception server closed the connection unexpectedly#012#011This probably means the server terminated abnormally#012#011before or while processing the request.

Mar 10 00:31:49 owamp-ps safe_run/scheduler ERROR    Exception: DatabaseError: server closed the connection unexpectedly#012#011This probably means the server terminated abnormally#012#011before or while processing the request.#012#012Traceback (most recent call last):#012  

File "/usr/lib/python2.6/site-packages/pscheduler/saferun.py", line 72, in safe_run#012    function()#012  

File "/usr/libexec/pscheduler/daemons/scheduler", line 809, in <lambda>#012    pscheduler.safe_run(lambda: main_program())#012  

File "/usr/libexec/pscheduler/daemons/scheduler", line 764, in main_program#012    cursor.execute(query, args)#012DatabaseError: server closed the connection unexpectedly#012#011This probably means the server terminated abnormally#012#011before or while processing the request.

Mar 10 00:31:49 owamp-ps safe_run/scheduler ERROR    Waiting 6.5 seconds before restarting

Mar 10 00:31:49 owamp-ps safe_run/archiver ERROR    Program threw an exception after 5:52:07.158001

Mar 10 00:31:49 owamp-ps safe_run/archiver ERROR    Exception: DatabaseError: server closed the connection unexpectedly#012#011This probably means the server terminated abnormally#012#011before or while processing the request.#012#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/pscheduler/saferun.py", line 72, in safe_run#012    function()#012  File "/usr/libexec/pscheduler/daemons/archiver", line 855, in <lambda>#012    pscheduler.safe_run(lambda: main_program())#012  File "/usr/libexec/pscheduler/daemons/archiver", line 811, in main_program#012    [options.max_parallel])#012  File "/usr/lib/python2.6/site-packages/pscheduler/db.py", line 188, in query#012    cursor.execute(query, args)#012DatabaseError: server closed the connection unexpectedly#012#011This probably means the server terminated abnormally#012#011before or while processing the request.

Mar 10 00:31:55 owamp-ps safe_run/scheduler ERROR    Exception: OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" and accepting#012#011TCP/IP connections on port 5432?#012#012Traceback (most recent call last):#012  

File "/usr/lib/python2.6/site-packages/pscheduler/saferun.py", line 72, in safe_run#012    function()#012  

File "/usr/libexec/pscheduler/daemons/scheduler", line 809, in <lambda>#012    pscheduler.safe_run(lambda: main_program())#012  

File "/usr/libexec/pscheduler/daemons/scheduler", line 734, in main_program#012    pg = pscheduler.pg_connection(dsn)#012  

File "/usr/lib/python2.6/site-packages/pscheduler/db.py", line 36, in pg_connection#012    pg = psycopg2.connect(dsn)#012  

File "/usr/lib64/python2.6/site-packages/psycopg2/__init__.py", line 164, in connect#012    conn = _connect(dsn, connection_factory=connection_factory, async=async)#012OperationalError: could not connect to server: Connection refused#012#011Is the server running on host "127.0.0.1" and accepting#012#011TCP/IP connections on port 5432?

Unsure about the first error but for the last one, postgres is running and listening to 5432 on localhost:

postgres   3159  0.0  0.0 341220 26348 ?        S    00:34   0:02 /usr/pgsql-9.5/bin/postmaster -D /var/lib/pgsql/9.5/data

postgres   3161  0.0  0.0 178392  1260 ?        Ss   00:34   0:00 postgres: logger process                                

postgres   3163  0.0  0.0 341324  4492 ?        Ss   00:34   0:00 postgres: checkpointer process                          

postgres   3164  0.0  0.0 341220  2460 ?        Ss   00:34   0:00 postgres: writer process                                

postgres   3165  0.0  0.0 341220  5620 ?        Ss   00:34   0:00 postgres: wal writer process                            

postgres   3166  0.0  0.0 341924  2428 ?        Ss   00:34   0:00 postgres: autovacuum launcher process                   

postgres   3167  0.0  0.0 178668  1612 ?        Ss   00:34   0:01 postgres: stats collector process                       

postgres  16623  0.0  0.0 346432 15224 ?        Ss   10:44   0:00 postgres: pscheduler pscheduler 127.0.0.1(50864) idle   

postgres  16624  0.0  0.0 342448  4480 ?        Ss   10:44   0:00 postgres: pscheduler pscheduler 127.0.0.1(50866) idle   

postgres  16678  0.0  0.0 345688  8108 ?        Ss   10:44   0:00 postgres: pscheduler pscheduler 127.0.0.1(50878) idle   

postgres  16679  0.0  0.0 342324  3380 ?        Ss   10:44   0:00 postgres: pscheduler pscheduler 127.0.0.1(50880) idle   

postgres  16680  0.0  0.0 342324  3388 ?        Ss   10:44   0:00 postgres: pscheduler pscheduler 127.0.0.1(50882) idle   

postgres  16721  0.0  0.0 345276  8840 ?        Ss   10:44   0:00 postgres: pscheduler pscheduler 127.0.0.1(50884) idle   

postgres  16722  0.0  0.0 342432  3760 ?        Ss   10:44   0:00 postgres: pscheduler pscheduler 127.0.0.1(50886) idle   

postgres  17083  0.0  0.1 531840 36996 ?        Ssl  10:45   0:00 postgres: pscheduler pscheduler 127.0.0.1(50902) idle   

postgres  17084  0.0  0.0 344792  6372 ?        Ss   10:45   0:00 postgres: pscheduler pscheduler 127.0.0.1(50903) idle   

postgres  39530 14.0  0.0 527504 24964 ?        Ssl  11:48   0:00 postgres: pscheduler pscheduler 127.0.0.1(56826) SELECT 

postgres  39555 13.0  0.0 527508 25000 ?        Ssl  11:48   0:00 postgres: pscheduler pscheduler 127.0.0.1(56840) SELECT 

postgres  39589 14.0  0.0 547400 23260 ?        Sl   11:48   0:00 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/tool/bwctliperf3/can-run

postgres  39649  0.0  0.0  94632  7324 ?        R    11:48   0:00 /usr/bin/python /usr/libexec/pscheduler/commands/../classes/tool/tracepath/can-run

tcp        0      0 127.0.0.1:5432              0.0.0.0:*                   LISTEN      3159/postmaster

I am assuming this error is one of the reason why there are no recent tests appearing within our PerfSONAR Dashboard (since Mar 4 2018).

Would anyone be able to help me with this issue?

Cheers,

Simon Green

SingAREN

PerfSONAR Node details: