perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

I think I found the problem. A restore on another machine fail and give a 
more specific error:

Restoring pScheduler from backup:
  Unpacking backup... Done.
  Restoring configuration... Done.
  Restoring database... Done.
  Updating database... Done.
  Resetting password... Done.
/usr/lib/perfsonar/scripts/ps-migrate-restore.sh: line 244: syntax error 
near unexpected token `fi'
/usr/lib/perfsonar/scripts/ps-migrate-restore.sh: line 244: `fi'

In   /usr/lib/perfsonar/scripts/ps-migrate-restore.sh
I added "then" to lines 235 and 245, reran restore, and it seems to have 
worked.

Kathy


On 8/23/2018 4:09 PM, Kathy Benninger wrote:
> I'm seeking advice for a pscheduler that did not survive data restore 
> following a Netinstall CentOS6->7 / pSTK4.1 Toolkit upgrade.
>
> The host doesn't initiate tests or respond to pscheduled tests from other 
> hosts.
>
> Following are test output and log file snippets that will hopefully be 
> sufficient to identify the problem.
>
> Thanks!
> Kathy Benninger
> Pittsburgh Supercomputing Center
>
> ---------------------------------------------------------------------------
> Some pscheduler checks:
>
> [root@ps benninge]# pscheduler ping localhost
> localhost: pScheduler is alive
> [root@ps benninge]# pscheduler troubleshoot
> Performing basic troubleshooting of localhost.
>
> localhost:
>
>   Checking path MTU... 65520+
>   Checking for pScheduler... OK.
>   Checking clock... OK.
>   Idle test... Failed.
> Unable to post task: Internal problem; see system logs.
>
> Nothing is being scheduled.
>
> ---------------------------------------------------------------------------
> pscheduler log messages:
>
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR    Exception: 
> ProgrammingError: function api_task_post(unknown, text[], unknown, 
> unknown, integer, unknown, boolean, unknown) does not exist
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"created-by": {"...
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR    Traceback (most recent 
> call last):
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
> full_dispatch_request
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR        rv = 
> self.dispatch_request()
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in 
> dispatch_request
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR        return 
> self.view_functions[rule.endpoint](**req.view_args)
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/tasks.py", line 477, in tasks
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR ProgrammingError: 
> function api_task_post(unknown, text[], unknown, unknown, integer, 
> unknown, boolean, unknown) does not exist
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"created-by": {"...
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:32:14 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR    Exception: 
> ProgrammingError: function api_task_post(unknown, text[], unknown, 
> unknown, integer, unknown, boolean, unknown) does not exist
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"psconfig": {"cr...
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR    Traceback (most recent 
> call last):
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
> full_dispatch_request
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR        rv = 
> self.dispatch_request()
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in 
> dispatch_request
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR        return 
> self.view_functions[rule.endpoint](**req.view_args)
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/tasks.py", line 477, in tasks
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR ProgrammingError: 
> function api_task_post(unknown, text[], unknown, unknown, integer, 
> unknown, boolean, unknown) does not exist
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"psconfig": {"cr...
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:32:38 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR    Exception: 
> ProgrammingError: function api_task_post(unknown, text[], unknown, 
> unknown, integer, unknown, boolean, unknown) does not exist
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"psconfig": {"cr...
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR    Traceback (most recent 
> call last):
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
> full_dispatch_request
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR        rv = 
> self.dispatch_request()
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in 
> dispatch_request
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR        return 
> self.view_functions[rule.endpoint](**req.view_args)
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/tasks.py", line 678, in tasks_uuid
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR ProgrammingError: 
> function api_task_post(unknown, text[], unknown, unknown, integer, 
> unknown, boolean, unknown) does not exist
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"psconfig": {"cr...
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:32:53 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR    Exception: 
> ProgrammingError: function api_task_post(unknown, text[], unknown, 
> unknown, integer, unknown, boolean, unknown) does not exist
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"psconfig": {"cr...
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR    Traceback (most recent 
> call last):
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
> full_dispatch_request
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR        rv = 
> self.dispatch_request()
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in 
> dispatch_request
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR        return 
> self.view_functions[rule.endpoint](**req.view_args)
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/tasks.py", line 477, in tasks
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/dbcursor.py", line 112, in dbcursor_query
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR ProgrammingError: 
> function api_task_post(unknown, text[], unknown, unknown, integer, 
> unknown, boolean, unknown) does not exist
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR    LINE 1: SELECT * FROM 
> api_task_post('{"reference": {"psconfig": {"cr...
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR                          ^
> Aug 23 14:33:04 ps journal: pscheduler-api ERROR    HINT:  No function 
> matches the given name and argument types. You might need to add explicit 
> type casts.
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Restarting: 
> ['/usr/libexec/pscheduler/daemons/archiver', '--daemon', '--pid-file', 
> '/run/pscheduler-archiver.pid', '--dsn', 
> '@/etc/pscheduler/database/database-dsn']
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Program threw an 
> exception after 0:00:00.012042
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Exception: 
> ProgrammingError: column "task_detail" does not exist
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    LINE 4: 
> ...               attempts, last_attempt, transform, task_detai...
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR ^
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Traceback (most 
> recent call last):
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in 
> safe_run
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR function()
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/libexec/pscheduler/daemons/archiver", line 879, in <lambda>
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR 
> pscheduler.safe_run(lambda: main_program())
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/libexec/pscheduler/daemons/archiver", line 835, in main_program
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR [options.max_parallel])
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/lib/python2.7/site-packages/pscheduler/db.py", line 193, in query
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR cursor.execute(query, 
> args)
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR ProgrammingError: 
> column "task_detail" does not exist
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    LINE 4: 
> ...               attempts, last_attempt, transform, task_detai...
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR ^
> Aug 23 14:33:12 ps journal: safe_run/archiver ERROR    Waiting 60.0 
> seconds before restarting
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Restarting: 
> ['/usr/libexec/pscheduler/daemons/archiver', '--daemon', '--pid-file', 
> '/run/pscheduler-archiver.pid', '--dsn', 
> '@/etc/pscheduler/database/database-dsn']
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Program threw an 
> exception after 0:00:00.012356
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Exception: 
> ProgrammingError: column "task_detail" does not exist
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    LINE 4: 
> ...               attempts, last_attempt, transform, task_detai...
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR ^
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Traceback (most 
> recent call last):
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 72, in 
> safe_run
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR function()
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/libexec/pscheduler/daemons/archiver", line 879, in <lambda>
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR 
> pscheduler.safe_run(lambda: main_program())
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/libexec/pscheduler/daemons/archiver", line 835, in main_program
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR [options.max_parallel])
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR      File 
> "/usr/lib/python2.7/site-packages/pscheduler/db.py", line 193, in query
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR cursor.execute(query, 
> args)
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR ProgrammingError: 
> column "task_detail" does not exist
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    LINE 4: 
> ...               attempts, last_attempt, transform, task_detai...
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR ^
> Aug 23 14:34:12 ps journal: safe_run/archiver ERROR    Waiting 60.0 
> seconds before restarting
>
> ---------------------------------------------------------------------------
> The first signs of trouble from when I tried to restore the data:
>
> [root@ps BACKUPS]# /usr/lib/perfsonar/scripts/ps-migrate-restore.sh --data 
> ps-backup_purdue_21Aug18.tgz
> WARN: Looks like non-root user accounts were created prior to running this 
> script. Skipping user account restoration to avoid conflicts
> Restoring perfsonar configuration...[SUCCESS]
> Restoring owamp-server configuration...[SUCCESS]
> Restoring NTP configuration...[SUCCESS]
> Restoring cassandra data for esmond...[SUCCESS]
> Restoring postgresql data for esmond...[SUCCESS]
>
> WARNING
>
> You are about to obliterate the contents of the pScheduler
> configuration and database on this host.  Please confirm this by
> typing today's date in YYYY-MM-DD format at the prompt below.
>
> Confirm (YYYY-MM-DD): 2018-08-22
>
> Restoring pScheduler from backup:
>   Unpacking backup... Done.
>   Restoring configuration... Done.
>   Restoring database... Done.
>   Updating database...ERROR:  deadlock detected
> DETAIL:  Process 20475 waits for AccessExclusiveLock on relation 36307 of 
> database 35991; blocked by process 20520.
> Process 20520 waits for AccessShareLock on relation 36229 of database 
> 35991; blocked by process 20475.
> HINT:  See server log for query details.
> CONTEXT:  SQL statement "DROP FUNCTION task_next_run(timestamptz, 
> timestamptz, interval) CASCADE;"
> PL/pgSQL function drop_function_all(text) line 34 at EXECUTE
> SQL statement "SELECT drop_function_all('task_next_run')"
> PL/pgSQL function inline_code_block line 1 at PERFORM
>
> WARNING:  Because of errors, pScheduler may be in an inoperable state.
> Unable to restore pScheduler configuration
> [root@ps BACKUPS]#
>
> [root@ps BACKUPS]# /usr/lib/perfsonar/scripts/ps-migrate-restore.sh --data 
> ps-backup_purdue_21Aug18.tgz
> WARN: Looks like non-root user accounts were created prior to running this 
> script. Skipping user account restoration to avoid conflicts
> Restoring perfsonar configuration...[SUCCESS]
> Restoring owamp-server configuration...[SUCCESS]
> Restoring NTP configuration...[SUCCESS]
> Restoring cassandra data for esmond...[SUCCESS]
> Restoring postgresql data for esmond...[SUCCESS]
>
> WARNING
>
> You are about to obliterate the contents of the pScheduler
> configuration and database on this host.  Please confirm this by
> typing today's date in YYYY-MM-DD format at the prompt below.
>
> Confirm (YYYY-MM-DD): 2018-08-22
>
> Restoring pScheduler from backup:
>   Unpacking backup... Done.
>   Restoring configuration... Done.
>   Restoring database... Done.
>   Updating database...ERROR:  deadlock detected
> DETAIL:  Process 9315 waits for AccessExclusiveLock on relation 60308 of 
> database 60105; blocked by process 9340.
> Process 9340 waits for AccessShareLock on relation 60316 of database 
> 60105; blocked by process 9315.
> HINT:  See server log for query details.
>
> WARNING:  Because of errors, pScheduler may be in an inoperable state.
> Unable to restore pScheduler configuration