perfSONAR User Q&A and Other Discussion

[Ricard Cruz <>]

Dear all,

This morning, after an automatic upgrade of our PerfSONAR installation to version 5.2.1 pScheduler are no longer working correctly.

Issues Observed:

• Database Constraint Violation

  
  

  ERROR: check constraint "update_times_nullness" of relation "run_state" is violated by some row

Jul  2 06:26:54 picperfsonar db-update-pscheduler[1175013]: ERROR:  check constraint "update_times_nullness" of relation "run_state" is violated by some row
Jul  2 06:26:54 picperfsonar rpm-post-wrapper[1175017]: ERROR:  check constraint "update_times_nullness" of relation "run_state" is violated by some row
Jul  2 06:26:58 picperfsonar /usr/sbin/httpd[669807]: pscheduler-api[669807] ERROR    Internal error 2001:67c:1148:204::250 POST https://picperfsonar/pscheduler/tasks: Error getting to
ols from lhcb-perf.nipne.ro: Failed to connect to lhcb-perf.nipne.ro port 443: Connection timed out
Jul  2 06:27:00 picperfsonar python3[1175178]: metrics ERROR    Couldn't collect stat for paused/ API return code:400, message: ion timed out after 5001 milliseconds with 0 bytes receive
Jul  2 06:27:05 picperfsonar python3[1175178]: metrics ERROR    Couldn't collect stat for archiving/backlog API return code:400, message: ion timed out after 5000 milliseconds with 0 bytes receive
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR    Exception: psycopg2.OperationalError: fe_sendauth: no password supplied
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR    Traceback (most recent call last):
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/usr/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        rv = self.dispatch_request()
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/usr/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/var/www/pscheduler-server/pschedulerapiserver/stat.py", line 65, in stat_archive_backlog
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        return single_numeric_query("""SELECT COUNT(*) FROM archiving
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/var/www/pscheduler-server/pschedulerapiserver/stat.py", line 48, in single_numeric_query
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        cursor = dbcursor_query(query, query_args)
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/var/www/pscheduler-server/pschedulerapiserver/dbcursor.py", line 99, in dbcursor_query
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        cursor = dbcursor()
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/var/www/pscheduler-server/pschedulerapiserver/dbcursor.py", line 82, in dbcursor
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        return cursor.cursor()
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/var/www/pscheduler-server/pschedulerapiserver/dbcursor.py", line 64, in cursor
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        raise reason;
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/var/www/pscheduler-server/pschedulerapiserver/dbcursor.py", line 54, in cursor
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        self.db = pscheduler.pg_connection(module.dsn, name="api")
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/usr/lib/python3.9/site-packages/pscheduler/db.py", line 43, in pg_connection
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        pg = psycopg2.connect(dsn)
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR      File "/usr/lib64/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR        conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR    psycopg2.OperationalError: fe_sendauth: no password supplied
Jul  2 06:27:05 picperfsonar /usr/sbin/httpd[1176839]: pscheduler-api[1176839] ERROR    Internal error 2001:67c:1148:204::250 GET https://picperfsonar/pscheduler/stat/archiving/backlog
: Internal problem; see system logs.

• pScheduler Scheduler Fails to Start due database error connection

Jul 02 08:38:41 picperfsonar /usr/sbin/httpd[1627]: pscheduler-api[1627] WARNING  Failed to connect to the database.
Jul 02 08:38:41 picperfsonar /usr/sbin/httpd[1627]: pscheduler-api[1627] ERROR    Exception: psycopg2.OperationalError: fe_sendauth: no password supplied
Jul 02 08:38:41 picperfsonar /usr/sbin/httpd[1627]: pscheduler-api[1627] ERROR    Traceback (most recent call last):

# systemctl status pscheduler-scheduler
× pscheduler-scheduler.service - pScheduler server - scheduler
     Loaded: loaded (/usr/lib/systemd/system/pscheduler-scheduler.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2025-07-02 08:45:40 CEST; 1h 48min ago
   Duration: 170ms
   Main PID: 12817 (code=exited, status=1/FAILURE)
        CPU: 175ms

Jul 02 08:45:40 picperfsonar systemd[1]: pscheduler-scheduler.service: Scheduled restart job, restart counter is at 5.
Jul 02 08:45:40 picperfsonar systemd[1]: Stopped pScheduler server - scheduler.
Jul 02 08:45:40 picperfsonar systemd[1]: pscheduler-scheduler.service: Start request repeated too quickly.
Jul 02 08:45:40 picperfsonar systemd[1]: pscheduler-scheduler.service: Failed with result 'exit-code'.
Jul 02 08:45:40 picperfsonar systemd[1]: Failed to start pScheduler server - scheduler.

• Database Seems to Be Running Fine

# systemctl status postgresql.service
● postgresql.service - PostgreSQL database server
     Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; preset: disabled)
     Active: active (running) since Wed 2025-07-02 08:33:12 CEST; 11min ago
   Main PID: 1538 (postmaster)
      Tasks: 8 (limit: 819986)
     Memory: 46.4M
        CPU: 8.471s
     CGroup: /system.slice/postgresql.service
             ├─1538 /usr/bin/postmaster -D /var/lib/pgsql/data
             ├─1558 "postgres: logger "
             ├─1581 "postgres: checkpointer "
             ├─1582 "postgres: background writer "
             ├─1584 "postgres: walwriter "
             ├─1585 "postgres: autovacuum launcher "
             ├─1586 "postgres: stats collector "
             └─1587 "postgres: logical replication launcher "

Jul 02 08:33:12 picperfsonar systemd[1]: Starting PostgreSQL database server...
Jul 02 08:33:12 picperfsonar postmaster[1538]: 2025-07-02 08:33:12.473 CEST [1538] LOG:  redirecting log output to logging collector process
Jul 02 08:33:12 picperfsonar postmaster[1538]: 2025-07-02 08:33:12.473 CEST [1538] HINT:  Future log output will appear in directory "log".
Jul 02 08:33:12 picperfsonar systemd[1]: Started PostgreSQL database server.

It appears that the upgrade may have introduced changes to the database schema or authentication handling that prevent pScheduler from functioning properly.

Would you please assist us in troubleshooting this? Let us know if you need any additional logs or information.

Thank you in advance for your support.

Best regards,

--

-----------------------------------------------------------------------
Ricard Cruz
Port d'Informació Científica (PIC)              e_mail:
Campus UAB - Edificio D                         Tel: +34 93 170 27 28
08193 Bellaterra (Barcelona), Spain      
http://www.pic.es
Avis - Aviso - Legal Notice: http://www.ifae.es/legal.html       
-------------------------------------------------------------------------