perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

I'm seeking advice for a pscheduler that did not survive data restore 
following a Netinstall CentOS6->7 / pSTK4.1 Toolkit upgrade.

The host doesn't initiate tests or respond to pscheduled tests from other 
hosts.

Following are test output and log file snippets that will hopefully be 
sufficient to identify the problem.

Thanks!
Kathy Benninger
Pittsburgh Supercomputing Center

---------------------------------------------------------------------------
Some pscheduler checks:

[root@ps benninge]# pscheduler ping localhost
localhost: pScheduler is alive
[root@ps benninge]# pscheduler troubleshoot
Performing basic troubleshooting of localhost.

localhost:

  Checking path MTU... 65520+
  Checking for pScheduler... OK.
  Checking clock... OK.
  Idle test... Failed.
Unable to post task: Internal problem; see system logs.

Nothing is being scheduled.

---------------------------------------------------------------------------
pscheduler log messages:

Aug 23 14:32:14 ps journal: pscheduler-api ERROR    Exception: 
ProgrammingError: function api_task_post(unknown, text[], unknown, unknown, 
integer, unknown, boolean, unknown) does not exist
Aug 23 14:32:14 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"created-by": {"...
Aug 23 14:32:14 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:32:14 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:32:14 ps journal: pscheduler-api ERROR    Traceback (most recent 
call last):
Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
full_dispatch_request
Aug 23 14:32:14 ps journal: pscheduler-api ERROR        rv = 
self.dispatch_request()
Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
Aug 23 14:32:14 ps journal: pscheduler-api ERROR        return 
self.view_functions[rule.endpoint](**req.view_args)
Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/tasks.py", line 477, in tasks
Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
Aug 23 14:32:14 ps journal: pscheduler-api ERROR ProgrammingError: function 
api_task_post(unknown, text[], unknown, unknown, integer, unknown, boolean, 
unknown) does not exist
Aug 23 14:32:14 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"created-by": {"...
Aug 23 14:32:14 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:32:14 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:32:38 ps journal: pscheduler-api ERROR    Exception: 
ProgrammingError: function api_task_post(unknown, text[], unknown, unknown, 
integer, unknown, boolean, unknown) does not exist
Aug 23 14:32:38 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"psconfig": {"cr...
Aug 23 14:32:38 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:32:38 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:32:38 ps journal: pscheduler-api ERROR    Traceback (most recent 
call last):
Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
full_dispatch_request
Aug 23 14:32:38 ps journal: pscheduler-api ERROR        rv = 
self.dispatch_request()
Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
Aug 23 14:32:38 ps journal: pscheduler-api ERROR        return 
self.view_functions[rule.endpoint](**req.view_args)
Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/tasks.py", line 477, in tasks
Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
Aug 23 14:32:38 ps journal: pscheduler-api ERROR ProgrammingError: function 
api_task_post(unknown, text[], unknown, unknown, integer, unknown, boolean, 
unknown) does not exist
Aug 23 14:32:38 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"psconfig": {"cr...
Aug 23 14:32:38 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:32:38 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:32:53 ps journal: pscheduler-api ERROR    Exception: 
ProgrammingError: function api_task_post(unknown, text[], unknown, unknown, 
integer, unknown, boolean, unknown) does not exist
Aug 23 14:32:53 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"psconfig": {"cr...
Aug 23 14:32:53 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:32:53 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:32:53 ps journal: pscheduler-api ERROR    Traceback (most recent 
call last):
Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
full_dispatch_request
Aug 23 14:32:53 ps journal: pscheduler-api ERROR        rv = 
self.dispatch_request()
Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
Aug 23 14:32:53 ps journal: pscheduler-api ERROR        return 
self.view_functions[rule.endpoint](**req.view_args)
Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/tasks.py", line 678, in tasks_uuid
Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
Aug 23 14:32:53 ps journal: pscheduler-api ERROR ProgrammingError: function 
api_task_post(unknown, text[], unknown, unknown, integer, unknown, boolean, 
unknown) does not exist
Aug 23 14:32:53 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"psconfig": {"cr...
Aug 23 14:32:53 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:32:53 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:33:04 ps journal: pscheduler-api ERROR    Exception: 
ProgrammingError: function api_task_post(unknown, text[], unknown, unknown, 
integer, unknown, boolean, unknown) does not exist
Aug 23 14:33:04 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"psconfig": {"cr...
Aug 23 14:33:04 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:33:04 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:33:04 ps journal: pscheduler-api ERROR    Traceback (most recent 
call last):
Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
full_dispatch_request
Aug 23 14:33:04 ps journal: pscheduler-api ERROR        rv = 
self.dispatch_request()
Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
"/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
Aug 23 14:33:04 ps journal: pscheduler-api ERROR        return 
self.view_functions[rule.endpoint](**req.view_args)
Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/tasks.py", line 477, in tasks
Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
"./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
Aug 23 14:33:04 ps journal: pscheduler-api ERROR ProgrammingError: function 
api_task_post(unknown, text[], unknown, unknown, integer, unknown, boolean, 
unknown) does not exist
Aug 23 14:33:04 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
api_task_post('{"reference": {"psconfig": {"cr...
Aug 23 14:33:04 ps journal: pscheduler-api ERROR                          ^
Aug 23 14:33:04 ps journal: pscheduler-api ERROR    HINT:  No function 
matches the given name and argument types. You might need to add explicit 
type casts.
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Restarting: 
['/usr/libexec/pscheduler/daemons/archiver', '--daemon', '--pid-file', 
'/run/pscheduler-archiver.pid', '--dsn', 
'@/etc/pscheduler/database/database-dsn']
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Program threw an 
exception after 0:00:00.012042
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Exception: 
ProgrammingError: column "task_detail" does not exist
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    LINE 4: 
...               attempts, last_attempt, transform, task_detai...
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR ^
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Traceback (most 
recent call last):
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR function()
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
"/usr/libexec/pscheduler/daemons/archiver", line 879, in <lambda>
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR 
pscheduler.safe_run(lambda: main_program())
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
"/usr/libexec/pscheduler/daemons/archiver", line 835, in main_program
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR [options.max_parallel])
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
"/usr/lib/python2.7/site-packages/pscheduler/db.py", line 193, in query
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR cursor.execute(query, 
args)
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR ProgrammingError: column 
"task_detail" does not exist
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    LINE 4: 
...               attempts, last_attempt, transform, task_detai...
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR ^
Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Waiting 60.0 seconds 
before restarting
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Restarting: 
['/usr/libexec/pscheduler/daemons/archiver', '--daemon', '--pid-file', 
'/run/pscheduler-archiver.pid', '--dsn', 
'@/etc/pscheduler/database/database-dsn']
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Program threw an 
exception after 0:00:00.012356
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Exception: 
ProgrammingError: column "task_detail" does not exist
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    LINE 4: 
...               attempts, last_attempt, transform, task_detai...
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR ^
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Traceback (most 
recent call last):
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in safe_run
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR function()
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
"/usr/libexec/pscheduler/daemons/archiver", line 879, in <lambda>
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR 
pscheduler.safe_run(lambda: main_program())
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
"/usr/libexec/pscheduler/daemons/archiver", line 835, in main_program
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR [options.max_parallel])
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
"/usr/lib/python2.7/site-packages/pscheduler/db.py", line 193, in query
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR cursor.execute(query, 
args)
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR ProgrammingError: column 
"task_detail" does not exist
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    LINE 4: 
...               attempts, last_attempt, transform, task_detai...
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR ^
Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Waiting 60.0 seconds 
before restarting

---------------------------------------------------------------------------
The first signs of trouble from when I tried to restore the data:

[root@ps BACKUPS]# /usr/lib/perfsonar/scripts/ps-migrate-restore.sh --data 
ps-backup_purdue_21Aug18.tgz
WARN: Looks like non-root user accounts were created prior to running this 
script. Skipping user account restoration to avoid conflicts
Restoring perfsonar configuration...[SUCCESS]
Restoring owamp-server configuration...[SUCCESS]
Restoring NTP configuration...[SUCCESS]
Restoring cassandra data for esmond...[SUCCESS]
Restoring postgresql data for esmond...[SUCCESS]

WARNING

You are about to obliterate the contents of the pScheduler
configuration and database on this host.  Please confirm this by
typing today's date in YYYY-MM-DD format at the prompt below.

Confirm (YYYY-MM-DD): 2018-08-22

Restoring pScheduler from backup:
  Unpacking backup... Done.
  Restoring configuration... Done.
  Restoring database... Done.
  Updating database...ERROR:  deadlock detected
DETAIL:  Process 20475 waits for AccessExclusiveLock on relation 36307 of 
database 35991; blocked by process 20520.
Process 20520 waits for AccessShareLock on relation 36229 of database 35991; 
blocked by process 20475.
HINT:  See server log for query details.
CONTEXT:  SQL statement "DROP FUNCTION task_next_run(timestamptz, 
timestamptz, interval) CASCADE;"
PL/pgSQL function drop_function_all(text) line 34 at EXECUTE
SQL statement "SELECT drop_function_all('task_next_run')"
PL/pgSQL function inline_code_block line 1 at PERFORM

WARNING:  Because of errors, pScheduler may be in an inoperable state.
Unable to restore pScheduler configuration
[root@ps BACKUPS]#

[root@ps BACKUPS]# /usr/lib/perfsonar/scripts/ps-migrate-restore.sh --data 
ps-backup_purdue_21Aug18.tgz
WARN: Looks like non-root user accounts were created prior to running this 
script. Skipping user account restoration to avoid conflicts
Restoring perfsonar configuration...[SUCCESS]
Restoring owamp-server configuration...[SUCCESS]
Restoring NTP configuration...[SUCCESS]
Restoring cassandra data for esmond...[SUCCESS]
Restoring postgresql data for esmond...[SUCCESS]

WARNING

You are about to obliterate the contents of the pScheduler
configuration and database on this host.  Please confirm this by
typing today's date in YYYY-MM-DD format at the prompt below.

Confirm (YYYY-MM-DD): 2018-08-22

Restoring pScheduler from backup:
  Unpacking backup... Done.
  Restoring configuration... Done.
  Restoring database... Done.
  Updating database...ERROR:  deadlock detected
DETAIL:  Process 9315 waits for AccessExclusiveLock on relation 60308 of 
database 60105; blocked by process 9340.
Process 9340 waits for AccessShareLock on relation 60316 of database 60105; 
blocked by process 9315.
HINT:  See server log for query details.

WARNING:  Because of errors, pScheduler may be in an inoperable state.
Unable to restore pScheduler configuration