perfSONAR User Q&A and Other Discussion

[Thomas Quentric <>]

Hello everyone,

After recently installing perfsonar, i went through a lot of error here and there so i followed the perfsonar troubleshooting guide (+using command : yum reinstall pscheduler server / systemctl restart pscheduler-scheduler ) to finally get this kind of error message on /var/log/perfsonar/pscheduler.log  through all my nodes :

Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    Program threw an exception after 0:00:04.458554
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    Program threw an exception after 0:00:02.537940
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    Program threw an exception after 0:00:02.206640
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    Exception: IntegrityError: duplicate key value violates unique constraint "heartbeat_pkey"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    DETAIL:  Key (name)=(scheduler) already exists.
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    CONTEXT:  SQL statement "INSERT INTO heartbeat (name, last) VALUES (new_name, now())"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    PL/pgSQL function heartbeat_boot(text) line 5 at SQL statement
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    Traceback (most recent call last):
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR      File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 73, in safe_run
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR        function()
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR      File "/usr/libexec/pscheduler/daemons/scheduler", line 923, in <lambda>
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR        pscheduler.safe_run(lambda: main_program())
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR      File "/usr/libexec/pscheduler/daemons/scheduler", line 816, in main_program
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR        cursor.execute("SELECT heartbeat_boot('scheduler')")
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    IntegrityError: duplicate key value violates unique constraint "heartbeat_pkey"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    DETAIL:  Key (name)=(scheduler) already exists.
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    CONTEXT:  SQL statement "INSERT INTO heartbeat (name, last) VALUES (new_name, now())"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    PL/pgSQL function heartbeat_boot(text) line 5 at SQL statement
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    Waiting 0.25 seconds before restarting
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    Exception: IntegrityError: duplicate key value violates unique constraint "heartbeat_pkey"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    DETAIL:  Key (name)=(runner) already exists.
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    CONTEXT:  SQL statement "INSERT INTO heartbeat (name, last) VALUES (new_name, now())"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    PL/pgSQL function heartbeat_boot(text) line 5 at SQL statement
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    Traceback (most recent call last):
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR      File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 73, in safe_run
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR        function()
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR      File "/usr/libexec/pscheduler/daemons/runner", line 957, in <lambda>
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR        pscheduler.safe_run(lambda: main_program())
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR      File "/usr/libexec/pscheduler/daemons/runner", line 769, in main_program
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR        cursor.execute("SELECT heartbeat_boot('runner')")
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    IntegrityError: duplicate key value violates unique constraint "heartbeat_pkey"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    DETAIL:  Key (name)=(runner) already exists.
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    CONTEXT:  SQL statement "INSERT INTO heartbeat (name, last) VALUES (new_name, now())"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    PL/pgSQL function heartbeat_boot(text) line 5 at SQL statement
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    Waiting 0.25 seconds before restarting
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    Exception: IntegrityError: duplicate key value violates unique constraint "heartbeat_pkey"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    DETAIL:  Key (name)=(archiver) already exists.
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    CONTEXT:  SQL statement "INSERT INTO heartbeat (name, last) VALUES (new_name, now())"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    PL/pgSQL function heartbeat_boot(text) line 5 at SQL statement
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    Traceback (most recent call last):
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR      File "/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 73, in safe_run
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR        function()
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR      File "/usr/libexec/pscheduler/daemons/archiver", line 930, in <lambda>
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR        pscheduler.safe_run(lambda: main_program())
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR      File "/usr/libexec/pscheduler/daemons/archiver", line 824, in main_program
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR        db.query("SELECT heartbeat_boot('archiver')")
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR      File "/usr/lib/python2.7/site-packages/pscheduler/db.py", line 193, in query
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR        cursor.execute(query, args)
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    IntegrityError: duplicate key value violates unique constraint "heartbeat_pkey"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    DETAIL:  Key (name)=(archiver) already exists.
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    CONTEXT:  SQL statement "INSERT INTO heartbeat (name, last) VALUES (new_name, now())"
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    PL/pgSQL function heartbeat_boot(text) line 5 at SQL statement
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    Waiting 0.25 seconds before restarting
Jun 25 14:24:57 pfs-sonde-cmc-i3-01 journal: safe_run/scheduler ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/scheduler', '--daemon', '--pid-file', '/run/pscheduler-server/pscheduler-scheduler.pid', '--dsn', '@/etc/pscheduler/d$
Jun 25 14:24:58 pfs-sonde-cmc-i3-01 journal: safe_run/runner ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/runner', '--daemon', '--pid-file', '/run/pscheduler-server/pscheduler-runner.pid', '--dsn', '@/etc/pscheduler/database/d$
Jun 25 14:24:58 pfs-sonde-cmc-i3-01 journal: safe_run/archiver ERROR    Restarting: ['/usr/libexec/pscheduler/daemons/archiver', '--daemon', '--pid-file', '/run/pscheduler-server/pscheduler-archiver.pid', '--dsn', '@/etc/pscheduler/data$

From there, it's kind of a dead end to me, so any help would be appreciated.

Thanks in advance,

Regards,

Thomas QUENTRIC