perfSONAR User Q&A and Other Discussion

[Petr Šesták <>]

Hello,

I did install perfSONAR Toolkit (Automated Script) on Alma 9 without 
problems. We have single physical host with a two separate Network 
Interface Card (NIC). The first is used for latency testing and the 
second is used for bandwidth. PerfSONAR is registered, Firewall is OK.

Services are:

[root@ps01-l pscheduler]# systemctl status pscheduler-scheduler
● pscheduler-scheduler.service - pScheduler server - scheduler
     Loaded: loaded 
(/usr/lib/systemd/system/pscheduler-scheduler.service; enabled; preset: 
disabled)
     Active: active (running) since Thu 2024-08-01 10:15:09 CEST; 4 
days ago
    Process: 3211518 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
    Process: 3211519 ExecStartPre=/bin/chmod 755 
/var/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
    Process: 3211520 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
    Process: 3211521 ExecStartPre=/bin/chmod 700 
/var/pscheduler-server/scheduler/tmp (code=exited, status=0/SUCCESS)
    Process: 3211522 ExecStartPre=/bin/chown -R pscheduler:pscheduler 
/var/pscheduler-server/scheduler (code=exited, status=0/SUCCESS)
    Process: 3211523 ExecStartPre=/bin/sh -c if [ -r 
/etc/pscheduler/daemons/scheduler.conf ]; then opts=$(sed -e 's/#.*$//' 
/etc/pscheduler/daemons/scheduler.conf); echo OPTIONS=$opts > 
/var/pscheduler-server/scheduler/options; chown pscheduler:pscheduler 
/var/pscheduler-server/scheduler/options; fi (code=exited, status=0/SUCCESS)
   Main PID: 3211526 (scheduler)
      Tasks: 2 (limit: 819046)
     Memory: 30.2M
        CPU: 25min 37.297s
     CGroup: /system.slice/pscheduler-scheduler.service
             └─3211526 /usr/bin/python3 
/usr/libexec/pscheduler/daemons/scheduler --dsn 
@/etc/pscheduler/database/database-dsn

Aug 05 10:27:54 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-05T19:21:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:28:41 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-05T19:46:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:31:17 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-05T21:51:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:31:55 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-05T22:16:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:32:32 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-05T22:41:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:33:07 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-05T23:06:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:33:49 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-05T23:31:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:34:59 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-06T00:21:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:36:28 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-06T01:36:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.
Aug 05 10:37:33 ps01-l.farm.particle.cz scheduler[2398130]: scheduler 
INFO     23438: Posting non-starting run at 2024-08-06T02:26:56+02:00 
for task e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51: Gave up after too many 
scheduling conflicts.

[root@ps01-l pscheduler]# systemctl status pscheduler-runner
● pscheduler-runner.service - pScheduler server - runner
     Loaded: loaded (/usr/lib/systemd/system/pscheduler-runner.service; 
enabled; preset: disabled)
     Active: active (running) since Thu 2024-08-01 10:15:37 CEST; 4 
days ago
    Process: 3211745 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/runner (code=exited, status=0/SUCCESS)
    Process: 3211746 ExecStartPre=/bin/chmod 755 
/var/pscheduler-server/runner (code=exited, status=0/SUCCESS)
    Process: 3211747 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/runner/tmp (code=exited, status=0/SUCCESS)
    Process: 3211748 ExecStartPre=/bin/chmod 700 
/var/pscheduler-server/runner/tmp (code=exited, status=0/SUCCESS)
    Process: 3211749 ExecStartPre=/bin/chown -R pscheduler:pscheduler 
/var/pscheduler-server/runner (code=exited, status=0/SUCCESS)
    Process: 3211750 ExecStartPre=/bin/sh -c if [ -r 
/etc/pscheduler/daemons/runner.conf ]; then opts=$(sed -e 's/#.*$//' 
/etc/pscheduler/daemons/runner.conf); echo OPTIONS=$opts > 
/var/pscheduler-server/runner/options; chown pscheduler:pscheduler 
/var/pscheduler-server/runner/options; fi (code=exited, status=0/SUCCESS)
   Main PID: 3211753 (runner)
      Tasks: 3 (limit: 819046)
     Memory: 34.5M
        CPU: 45min 38.673s
     CGroup: /system.slice/pscheduler-runner.service
             └─3211753 /usr/bin/python3 
/usr/libexec/pscheduler/daemons/runner --dsn 
@/etc/pscheduler/database/database-dsn

Aug 05 10:26:13 ps01-l.farm.particle.cz runner[2405952]: runner WARNING 
39765: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/6c36ec1f-e51a-4d6a-bb9f-540162709dcf/runs/c9b8849a-9e85-4765-b4c7-d6ae4e18cc1e: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:32:32 ps01-l.farm.particle.cz runner[2410894]: runner WARNING 
39312: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/e881fe33-a052-436e-af6a-d9bf6be9be3f/runs/7d3fd1f7-94a4-4d0b-bb7b-988f75725133: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:43:31 ps01-l.farm.particle.cz runner[2418304]: runner WARNING 
39514: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/5967f96c-d2ec-4509-8b5b-7d2403257222/runs/910bb08e-cdbc-48ea-9407-1043fe66b83a: 
400: Operation timed out after 14501 milliseconds with 0 bytes received
Aug 05 10:44:50 ps01-l.farm.particle.cz runner[2418304]: runner WARNING 
38957: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/3d0d1fc0-5759-40e9-ac01-032aa5561d0f/runs/28479001-a3c0-4008-9067-8ff09630e47c: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:47:32 ps01-l.farm.particle.cz runner[2420424]: runner WARNING 
39515: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/5967f96c-d2ec-4509-8b5b-7d2403257222/runs/cf5faf25-6bd0-46c9-a95b-9c03654d36a6: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:50:20 ps01-l.farm.particle.cz runner[2421922]: runner WARNING 
39079: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/59a2d715-010b-459a-8dcf-8d328b9f9d67/runs/fc4e7c59-36e2-4508-bcd9-a0d28ec3de49: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:50:54 ps01-l.farm.particle.cz runner[2421922]: runner WARNING 
39766: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/6c36ec1f-e51a-4d6a-bb9f-540162709dcf/runs/0df0b4e4-79af-4f47-b9a8-65b8ea42ed80: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:52:24 ps01-l.farm.particle.cz runner[2423165]: runner WARNING 
39313: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/e881fe33-a052-436e-af6a-d9bf6be9be3f/runs/87c053be-b9eb-47dc-9992-387edcdbdd9d: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:54:09 ps01-l.farm.particle.cz runner[2423993]: runner WARNING 
40703: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/197bf75b-7fb4-4d5c-9659-82a2c4f3c667/runs/e763d619-958e-4441-a310-33a435e10dff: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug 05 10:55:12 ps01-l.farm.particle.cz runner[2424756]: runner WARNING 
39080: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/59a2d715-010b-459a-8dcf-8d328b9f9d67/runs/b8bd8c35-b944-47be-b962-2fe718b7aad1: 
400: Operation timed out after 14501 milliseconds with 0 bytes received

[root@ps01-l pscheduler]# systemctl status pscheduler-archiver
● pscheduler-archiver.service - pScheduler server - archiver
     Loaded: loaded 
(/usr/lib/systemd/system/pscheduler-archiver.service; enabled; preset: 
disabled)
     Active: active (running) since Thu 2024-08-01 10:16:25 CEST; 4 
days ago
    Process: 3212225 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
    Process: 3212226 ExecStartPre=/bin/chmod 755 
/var/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
    Process: 3212227 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
    Process: 3212228 ExecStartPre=/bin/chmod 700 
/var/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
    Process: 3212229 ExecStartPre=/bin/chown -R pscheduler:pscheduler 
/var/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
    Process: 3212230 ExecStartPre=/bin/sh -c if [ -r 
/etc/pscheduler/daemons/archiver.conf ]; then opts=$(sed -e 's/#.*$//' 
/etc/pscheduler/daemons/archiver.conf); echo OPTIONS=$opts > 
/var/pscheduler-server/archiver/options; chown pscheduler:pscheduler 
/var/pscheduler-server/archiver/options; fi (code=exited, status=0/SUCCESS)
   Main PID: 3212233 (archiver)
      Tasks: 2 (limit: 819046)
     Memory: 32.0M
        CPU: 2min 23.227s
     CGroup: /system.slice/pscheduler-archiver.service
             └─3212233 /usr/bin/python3 
/usr/libexec/pscheduler/daemons/archiver --dsn 
@/etc/pscheduler/database/database-dsn

Aug 03 19:50:33 ps01-l.farm.particle.cz archiver[1055237]: archiver 
WARNING  3204: Failed to archive 
https://ps01-l.farm.particle.cz/pscheduler/tasks/e24f7abc-21d2-4cb3-88dc-67d30cc38989/runs/74f58e57-178b-4112-9485-13dff578d041 
to http: Possibly-permanent archiver error: Program failed to start 
after 3 tries: Timed out waiting for read
Aug 03 19:50:33 ps01-l.farm.particle.cz archiver[1055237]: archiver 
WARNING  3204: Gave up archiving 
https://ps01-l.farm.particle.cz/pscheduler/tasks/e24f7abc-21d2-4cb3-88dc-67d30cc38989/runs/74f58e57-178b-4112-9485-13dff578d041 
to http
Aug 04 11:57:03 ps01-l.farm.particle.cz archiver[1618436]: archiver 
WARNING  3301: Failed to archive 
https://ps01-l.farm.particle.cz/pscheduler/tasks/e24f7abc-21d2-4cb3-88dc-67d30cc38989/runs/975cb992-884b-4301-90be-3852b4dc2466 
to http: Possibly-permanent archiver error: Program failed to start 
after 3 tries: Timed out waiting for read
Aug 04 11:57:03 ps01-l.farm.particle.cz archiver[1618436]: archiver 
WARNING  3301: Gave up archiving 
https://ps01-l.farm.particle.cz/pscheduler/tasks/e24f7abc-21d2-4cb3-88dc-67d30cc38989/runs/975cb992-884b-4301-90be-3852b4dc2466 
to http
Aug 04 18:08:22 ps01-l.farm.particle.cz archiver[1835758]: archiver 
WARNING  3341: Failed to archive 
https://ps01-l.farm.particle.cz/pscheduler/tasks/30304843-f803-4277-9b23-207fefc535be/runs/f0f20f2d-9442-4f09-8285-5229bb0bc4ca 
to http: Possibly-permanent archiver error: Program failed to start 
after 3 tries: Timed out waiting for read
Aug 04 18:08:22 ps01-l.farm.particle.cz archiver[1835758]: archiver 
WARNING  3341: Gave up archiving 
https://ps01-l.farm.particle.cz/pscheduler/tasks/30304843-f803-4277-9b23-207fefc535be/runs/f0f20f2d-9442-4f09-8285-5229bb0bc4ca 
to http
Aug 05 00:58:04 ps01-l.farm.particle.cz archiver[2078287]: archiver 
WARNING  3381: Failed to archive 
https://ps01-l.farm.particle.cz/pscheduler/tasks/30304843-f803-4277-9b23-207fefc535be/runs/c5442846-1959-4953-a52f-80f02c73b2d2 
to http: Possibly-permanent archiver error: Program failed to start 
after 3 tries: Timed out waiting for read
Aug 05 00:58:04 ps01-l.farm.particle.cz archiver[2078287]: archiver 
WARNING  3381: Gave up archiving 
https://ps01-l.farm.particle.cz/pscheduler/tasks/30304843-f803-4277-9b23-207fefc535be/runs/c5442846-1959-4953-a52f-80f02c73b2d2 
to http
Aug 05 09:14:47 ps01-l.farm.particle.cz archiver[2362994]: archiver 
WARNING  3427: Failed to archive 
https://ps01-l.farm.particle.cz/pscheduler/tasks/bae65ed8-10cd-4d08-9d6b-c376485c9f80/runs/06fb553e-9ccb-4830-8a03-f009b732167c 
to http: Possibly-permanent archiver error: Program failed to start 
after 3 tries: Timed out waiting for read
Aug 05 09:14:47 ps01-l.farm.particle.cz archiver[2362994]: archiver 
WARNING  3427: Gave up archiving 
https://ps01-l.farm.particle.cz/pscheduler/tasks/bae65ed8-10cd-4d08-9d6b-c376485c9f80/runs/06fb553e-9ccb-4830-8a03-f009b732167c 
to http

[root@ps01-l pscheduler]# systemctl status pscheduler-ticker
● pscheduler-ticker.service - pScheduler server - ticker
     Loaded: loaded (/usr/lib/systemd/system/pscheduler-ticker.service; 
enabled; preset: disabled)
     Active: active (running) since Thu 2024-08-01 10:16:24 CEST; 4 
days ago
    Process: 3212100 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
    Process: 3212101 ExecStartPre=/bin/chmod 755 
/var/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
    Process: 3212102 ExecStartPre=/bin/mkdir -p 
/var/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
    Process: 3212103 ExecStartPre=/bin/chmod 700 
/var/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
    Process: 3212104 ExecStartPre=/bin/chown -R pscheduler:pscheduler 
/var/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
    Process: 3212105 ExecStartPre=/bin/sh -c if [ -r 
/etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' 
/etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > 
/var/pscheduler-server/ticker/options; chown pscheduler:pscheduler 
/var/pscheduler-server/ticker/options; fi (code=exited, status=0/SUCCESS)
   Main PID: 3212108 (ticker)
      Tasks: 2 (limit: 819046)
     Memory: 28.7M
        CPU: 20.340s
     CGroup: /system.slice/pscheduler-ticker.service
             └─3212108 /usr/bin/python3 
/usr/libexec/pscheduler/daemons/ticker --dsn 
@/etc/pscheduler/database/database-dsn

Aug 01 10:16:24 ps01-l.farm.particle.cz systemd[1]: Starting pScheduler 
server - ticker...
Aug 01 10:16:24 ps01-l.farm.particle.cz systemd[1]: Started pScheduler 
server - ticker.
Aug 01 10:16:24 ps01-l.farm.particle.cz ticker[3212108]: ticker INFO 
Started

[root@ps01-l pscheduler]# systemctl status psconfig-pscheduler-agent
● psconfig-pscheduler-agent.service - pSConfig PScheduler Agent
     Loaded: loaded 
(/usr/lib/systemd/system/psconfig-pscheduler-agent.service; enabled; 
preset: disabled)
     Active: active (running) since Fri 2024-07-26 06:38:12 CEST; 1 
week 3 days ago
   Main PID: 1982308 (psconfig_psched)
      Tasks: 1 (limit: 819046)
     Memory: 24.5M
        CPU: 27.543s
     CGroup: /system.slice/psconfig-pscheduler-agent.service
             └─1982308 /usr/bin/python3 
/usr/lib/perfsonar/psconfig/bin/psconfig_pscheduler_agent 
--config=/etc/perfsonar/psconfig/pscheduler-agent.json 
--logger=/etc/perfsonar/psconfig/pscheduler-agent-logger.conf

Jul 26 06:38:12 ps01-l.farm.particle.cz systemd[1]: Starting pSConfig 
PScheduler Agent...
Jul 26 06:38:12 ps01-l.farm.particle.cz systemd[1]: Started pSConfig 
PScheduler Agent.

[root@ps01-l pscheduler]# systemctl status owamp-server
● owamp-server.service - OWAMP server
     Loaded: loaded (/usr/lib/systemd/system/owamp-server.service; 
enabled; preset: disabled)
     Active: active (running) since Fri 2024-07-26 06:38:17 CEST; 1 
week 3 days ago
   Main PID: 1985540 (owampd)
      Tasks: 430 (limit: 819046)
     Memory: 469.1M
        CPU: 10h 6min 33.564s
     CGroup: /system.slice/owamp-server.service
             ├─ 341260 /usr/bin/owampd -c /etc/owamp-server -R /var/run
             ├─ 908502 /usr/bin/owampd -c /etc/owamp-server -R /var/run
             ├─1116734 /usr/bin/owampd -c /etc/owamp-server -R /var/run

[root@ps01-l pscheduler]# systemctl status perfsonar-lsregistrationdaemon
● perfsonar-lsregistrationdaemon.service - perfSONAR Lookup Service 
Registration Daemon
     Loaded: loaded 
(/usr/lib/systemd/system/perfsonar-lsregistrationdaemon.service; 
enabled; preset: disabled)
     Active: active (running) since Sun 2024-07-28 16:02:52 CEST; 1 
week 0 days ago
   Main PID: 107464 (lsregistrationd)
      Tasks: 1 (limit: 819046)
     Memory: 138.7M
        CPU: 8min 8.122s
     CGroup: /system.slice/perfsonar-lsregistrationdaemon.service
             └─107464 lsregistrationdaemon.pl

Aug 05 09:04:42 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in numeric ge (>=) at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 357.
Aug 05 09:04:45 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in hash element at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 473.
Aug 05 10:04:43 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in numeric ge (>=) at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 357.
Aug 05 10:04:43 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in numeric ge (>=) at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 357.
Aug 05 10:04:43 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in numeric ge (>=) at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 357.
Aug 05 10:04:43 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in numeric ge (>=) at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 357.
Aug 05 10:04:43 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in numeric ge (>=) at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 357.
Aug 05 10:04:44 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in numeric ge (>=) at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 357.
Aug 05 10:04:44 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in hash element at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 473.
Aug 05 10:04:44 ps01-l.farm.particle.cz lsregistrationdaemon.pl[107464]: 
Use of uninitialized value in hash element at 
/usr/lib/perfsonar/bin/../lib/perfSONAR_PS/LSRegistrationDaemon/Base.pm 
line 492.

ping from both interaces is OK.
ping -I eno1 8.8.8.8
ping -I enp23s0f1np1 8.8.8.8

Manually running tests with pscheduler for throughput, latency and rtt 
are OK.

Automatically measured throughput is mainly (only) in reverse measurments.

Web for our perfsonar is https://ps01-l.farm.particle.cz/

Checking pscheduler is OK:
pscheduler troubleshoot --host=ps01-l.farm.particle.cz --ip-version=4
pscheduler troubleshoot --host=ps01-l.farm.particle.cz --ip-version=6
pscheduler troubleshoot --host=ps02-b.farm.particle.cz --ip-version=4
pscheduler troubleshoot --host=ps02-b.farm.particle.cz --ip-version=6

A check of our perfsonar services shows that no meshes are configured 
and there is a problem with esmond:
https://psetf.opensciencegrid.org/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fhost%3Dps01-l.farm.particle.cz%26site%3Detf%26view_name%3Dhost
https://psetf.opensciencegrid.org/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fhost%3Dps02-b.farm.particle.cz%26site%3Detf%26view_name%3Dhost

/var/log/opensearch/opensearch.log:
[2024-08-06T09:40:43,703][WARN ][o.o.p.c.u.JsonConverter  ] 
[ps01-l.farm.particle.cz] Json Mapping Error: Cannot invoke 
"java.lang.Long.longValue()" because "this.cacheMaxSize" is null 
(through reference chain: 
org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])

/var/log/pscheduler/pscheduler.log:
Aug  6 08:27:34 ps01-l.farm.particle.cz scheduler[3167949]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T13:38:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:28:13 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T14:03:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:29:10 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T14:53:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:30:01 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T15:43:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:30:53 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T16:33:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:31:46 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T16:58:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:32:20 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T17:23:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:33:18 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T17:48:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:33:35 ps01-l.farm.particle.cz runner[3177389]: runner WARNING 
41266: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51/runs/c30db67a-b15a-4c55-b5b2-973b603d795b: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug  6 08:34:31 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T19:03:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:34:57 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T19:28:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:36:09 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T20:18:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:36:29 ps01-l.farm.particle.cz runner[3179927]: runner WARNING 
41896: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/edabfece-c86a-466f-846b-fb2b5084002a/runs/87ac299c-36fa-4ff2-a092-f5ca5fa0508e: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug  6 08:37:40 ps01-l.farm.particle.cz runner[3180613]: runner WARNING 
41685: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/9f0c5310-6721-4d22-bd08-a7e8ad3db393/runs/6d1201de-1b31-4b5f-ad60-f025042579c8: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug  6 08:38:03 ps01-l.farm.particle.cz scheduler[3173914]: scheduler 
INFO     23864: Posting non-starting run at 2024-08-06T22:48:35+02:00 
for task 681b55cc-2a4c-4da5-b838-262fd0d903a2: Gave up after too many 
scheduling conflicts.
Aug  6 08:39:54 ps01-l.farm.particle.cz runner[3182195]: runner WARNING 
41442: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/9c49174a-df51-472d-ba06-944104a0dc3a/runs/890b3f12-8d2b-4e04-b97a-3a72bfa0f20e: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug  6 08:41:51 ps01-l.farm.particle.cz runner[3183687]: runner WARNING 
41897: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/edabfece-c86a-466f-846b-fb2b5084002a/runs/b871ca47-03f8-4c58-8857-6b130f519560: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug  6 08:44:07 ps01-l.farm.particle.cz runner[3185119]: runner WARNING 
42252: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/ef8519c9-de0e-4361-92ff-3e69eacd1f3d/runs/70300c74-268f-41f4-8da3-f24a203081d8: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug  6 08:50:56 ps01-l.farm.particle.cz runner[3188519]: runner WARNING 
43081: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/e0511e76-cfd5-4503-bf5e-4ce4c06ef172/runs/cc3d9c05-2c35-4e06-9777-d611f3d12df4: 
400: Operation timed out after 14501 milliseconds with 0 bytes received
Aug  6 08:53:34 ps01-l.farm.particle.cz runner[3190243]: runner WARNING 
42253: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/ef8519c9-de0e-4361-92ff-3e69eacd1f3d/runs/3e23b70a-8607-4a7c-bbad-a676cf074029: 
400: Operation timed out after 14500 milliseconds with 0 bytes received
Aug  6 09:02:18 ps01-l.farm.particle.cz runner[3194955]: runner WARNING 
41267: Unable to retrieve run 
https://130.184.244.8/pscheduler/tasks/e68c4cfa-6133-48eb-8ed9-b4cfbddf5a51/runs/b9229e01-349f-45ca-bd55-d0cc4ed0e3d7: 
400: Operation timed out after 14500 milliseconds with 0 bytes received

Please can you help.

Peter Sestak

Attachment: smime.p7s
Description: Elektronicky podpis S/MIME