Any update on this issue?
From:
Wilson Cheung <>
Date: Monday, September 11, 2023 at 8:26 PM
To: Mark Chouteau <>, Horstmann, Ben <>
Cc: Garnizov, Ivan <>, perfsonar-user () <>, Fedorka, Shayne <>, Robert Lageano <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
CAUTION:
This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.
My mailer apparently line wrapped some of the earlier replies oddly on my monitor when I was reading it. The example "Failed to..." errors in "pscheduler.log" that
I referenced were actually in the email that was sent by Robert Lageano <> earlier, not the earlier email from Ivan Garnizov <>. Sorry for the confusion.
I'm going to try testing on a Alma/Rocky EL9 install next time to see if it makes any difference
From: Mark Chouteau <>
Sent: Monday, September 11, 2023 12:17 PM
To: Horstmann, Ben <>
Cc: Wilson Cheung <>; Garnizov, Ivan <>; perfsonar-user () <>; Fedorka, Shayne <>; Robert Lageano <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
I see this issue also, Building out new nodes in the last 3 weeks:
New CENT install is fine.
New installs using Ubuntu 20.04 on bare metal and VM both have the issue of not showing test results.
I have experienced the “Not Running” issue once.
On Sep 11, 2023, at 2:41 PM, Horstmann, Ben <> wrote:
I am experiencing the same behavior as Ivan and Wilson. Clean installs of Ubuntu and Alma experience this. Have not tried CENT.
We are using the traditional repo, not the early release. Are the fixes perhaps included in EL and not yet in prod? Thanks!
Assistant Director, Network Operations Center
Washington University in St. Louis
4480 Clayton Ave, St. Louis, MO 63110
From: <>
on behalf of Wilson Cheung <>
Reply-To: Wilson Cheung <>
Date: Monday, September 11, 2023 at 1:31 PM
To: "Garnizov, Ivan" <>, "perfsonar-user ()"
<>, "Fedorka, Shayne" <>, Robert Lageano
<>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
I'll chime in and say that I'm also experiencing the same problems with fresh installs of Ubuntu Server 20.04.6 with latest OS updates plus latest perfSONAR 5.0.4 Toolkit. I
did the "apt install perfsonar-toolkit" option as described at "https://docs.perfsonar.net/install_debian.html".
Nothing ever shows up in the "Test Results" section of the Toolkit webpage. Have tried on multiple Ubuntu installs on both physical and virtual machine servers with a mix of
1GigE/10GigE NICs. I'm able to login via the toolkit webpages and add more tests to run more frequently (e.g. every 15-minutes) to see if anything would show up.
I did a separate full pS 5.0.4 "toolkit" install on a fresh install of CentOS 7.9-2009 plus latest OS updates to compare and results do appear under the "Test Results" page,
so this problem may be Ubuntu specific. Haven't tried other Debian-based distros yet.
On the Toolkit webpage of the Ubuntu servers, it says "pscheduler" is "Running" in green. If I leave it in this state and don't reboot the servers for a few days, the pscheduler
status tends to eventually changes to "Not Running" in red. If I manually restart all of the pscheduler services, it goes back to "Running". Still no "Test Results" appear locally.
On the Ubuntu servers, I also see the same errors in the "pscheduler.log "that Ivan listed about "Failed to archive" and "Failed to put result:
503" and "503 Service Unavailable".
I also see other errors in "logstash-plain.log" and "opensearch.log" about Java. A recent post talked about making sure your servers are not under-resourced and allocating more
memory resources to the JVM used by opensearch/logstash.
Before I dig further down into the various pS internals, are there other specific logs or others I should focus on checking? I'm not sure if there's something pS configuration
specific I missed.
From: <>
on behalf of Robert Lageano <>
Sent: Sunday, September 10, 2023 11:57 PM
To: Garnizov, Ivan <>; perfsonar-user ()
<>; Fedorka, Shayne <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
I have a fresh install of Ubuntu Server 20.04 and Perfsonar Toolkit but it refuses to show the test results. Can someone help me with the following issue:
● pscheduler-archiver.service - pScheduler server - archiver
Loaded: loaded (/lib/systemd/system/pscheduler-archiver.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-09-11 01:08:16 UTC; 5h 47min ago
Process: 61558 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
Process: 61563 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
Process: 61567 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
Process: 61569 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
Process: 61571 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
Process: 61574 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
Process: 61577 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/archiver.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/archiver.conf); echo OPTIONS=$opts
> /var/run/pscheduler-server/archiver/options; chown pscheduler:pscheduler /var/run/pscheduler-server/archiver/options; fi (code=exited, status=0/SUCCESS)
Main PID: 61585 (python3)
CGroup: /system.slice/pscheduler-archiver.service
└─61585 python3 /usr/lib/pscheduler/daemons/archiver --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/archiver/pid
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING <html><head>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING <title>503 Service Unavailable</title>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING </head><body>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING <h1>Service Unavailable</h1>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING <p>The server is temporarily unable to service your
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING request due to maintenance downtime or capacity
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING problems. Please try again later.</p>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING </body></html>
-----------------------------------------------------------------------------
Robert Lageano
Senior IT Network Security Specialist
National Computational Infrastructure (NCI)
The Australian National University
143 Ward Road
Acton, ACT, 2601
M: +61 452 535 678
Want the latest news from NCI? nci.org.au
Find out more about NCI: YouTube | Facebook | Twitter | LinkedIn | Podcast
From: <>
on behalf of "Fedorka, Shayne" <>
Sent: Saturday, September 9, 2023 7:12 AM
To: Garnizov, Ivan <>; perfsonar-user ()
<>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
Thanks for looking into this Ivan. Something must have gone wrong with my upgrade from Ubuntu 18.04 to 20.04 and/or perfsonar v4 to v5.
I just did a clean install of 20.04 on the server followed by perfsonar v5 and all services are running without issue.
From: Garnizov, Ivan <>
Date: Friday, September 8, 2023 at 9:31 AM
To: Fedorka, Shayne <>, perfsonar-user ()
<>
Subject: RE: pscheduler service status (Not running) on Dashboard
CAUTION: This email originated from
outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I am not sure about the problem and the observation of Shayne, but…
I have completed a fresh install of the pS Testpoint 5.0.4 on Ubuntu 20.04 and have seen no issues.
@r72:~$ systemctl status pscheduler-ticker
в—Џ pscheduler-ticker.service - pScheduler server - ticker
Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-09-08 15:18:34 UTC; 7min ago
Main PID: 13392 (python3)
CGroup: /system.slice/pscheduler-ticker.service
в””в”Ђ13392 python3 /usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/ticker/pid
Sep 08 15:18:34 r72.rrze systemd[1]: Starting pScheduler server - ticker...
Sep 08 15:18:34 r72.rrze systemd[1]: Started pScheduler server - ticker.
Sep 08 15:18:34 r72.rrze python3[13392]: ticker INFO Started
@r72:~$ dpkg -l | grep pscheduler-server
ii pscheduler-server 5.0.4-1 all pScheduler server components
GEANT WP6T3: pS development team
GEANT WP9T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
From: Fedorka, Shayne []
Sent: Wednesday, September 6, 2023 5:25 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard
/usr/bin/python3: No code object available for pycurl
From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:24 AM
To: Fedorka, Shayne <>
Subject: RE: pscheduler service status (Not running) on Dashboard
CAUTION: This email originated from
outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.
From: Fedorka, Shayne []
Sent: Wednesday, September 6, 2023 5:17 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard
I just confirmed, that package is installed. Any other ideas?
apt list --installed | grep pycurl
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
python3-pycurl/focal,now 7.43.0.2-1ubuntu5 amd64 [installed]
From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:12 AM
To: Fedorka, Shayne <>
Cc: perfsonar-user ()
<>
Subject: RE: pscheduler service status (Not running) on Dashboard
CAUTION: This email originated from
outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.
The real problem is actually in this error message:
WARNING Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'
You are most likely somehow missing this package
All credits to Mark Feit for the information comes from him.
GEANT WP6T3: pS development team
GEANT WP9T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
From: Fedorka, Shayne []
Sent: Wednesday, September 6, 2023 5:05 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard
The output of the requested command does not return anything.
Yes, the one command did get cut, here is the full output:
Process: 576009 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pscheduler-server/ticker/options;
chown pscheduler:pscheduler /var
From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 6:51 AM
To: Fedorka, Shayne <>
Subject: RE: pscheduler service status (Not running) on Dashboard
CAUTION: This email originated from
outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Indeed this is a strange problem. I am myself running a test pS instance of Ubuntu 20.04.
Please share the output of this command:
sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf
Is there some cutting of text from the diagnostic window?
I am not sure this command from your output makes sense:
echo OPTIONS=$opts > /var/run/pschedule
GEANT WP6T3: pS development team
GEANT WP9T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
From: [] On
Behalf Of "Fedorka, Shayne"
Sent: Tuesday, September 5, 2023 10:00 PM
To:
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
I also have this issue after upgrading from perfsonar version 4 to 5 on Ubuntu 20.04. I found the pscheduler-ticker service is failing but I’m not sure why.
sudo systemctl status pscheduler-ticker.service
â— pscheduler-ticker.service - pScheduler server - ticker
Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2023-09-05 13:26:54 MDT; 2s ago
Process: 3284 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
Process: 3285 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
Process: 3286 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
Process: 3287 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
Process: 3288 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
Process: 3289 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
Process: 3290 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pschedule>
Process: 3293 ExecStart=/usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn $OPTIONS --pid-file /var/run/pscheduler-server/ticker/pid (code=exited, status=1/FAILURE)
Process: 3300 ExecStopPost=/bin/rm -rf /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
Main PID: 3293 (code=exited, status=1/FAILURE)
Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function warm_boot() line 4 at PERFORM
Sep 05 13:26:58 esif-perf05-10g ticker[3323]: SQL statement "SELECT warm_boot()"
Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function cold_boot() line 4 at PERFORM
Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.
Sep 05 13:27:01 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Scheduled restart job, restart counter is at 34.
Sep 05 13:27:01 esif-perf05-10g systemd[1]: Stopped pScheduler server - ticker.
Sep 05 13:27:01 esif-perf05-10g systemd[1]: Starting pScheduler server - ticker...
Sep 05 13:27:01 esif-perf05-10g systemd[1]: Started pScheduler server - ticker.
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker INFO Started
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING CONTEXT: Traceback (most recent call last):
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING PL/Python function "http_delete", line 3, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING import pscheduler
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING PL/Python function "http_delete", line 4, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING PL/Python function "http_delete", line 14, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING PL/Python function "http_delete", line 7, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING PL/Python function "http_delete"
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING PL/pgSQL function http_queue_process(bigint) line 19 at assignment
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING SQL statement "SELECT http_queue_process(id) FROM http_queue
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING WHERE next_attempt < now() OR attempts = 0"
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING PL/pgSQL function http_queue_process_all() line 3 at PERFORM
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: Traceback (most recent call last):
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: File "/usr/lib/pscheduler/daemons/ticker", line 172, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: main_program()
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: File "/usr/lib/pscheduler/daemons/ticker", line 132, in main_program
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: cursor.execute("SELECT cold_boot()")
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: psycopg2.errors.ExternalRoutineException: ModuleNotFoundError: No module named 'pycurl'
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: CONTEXT: Traceback (most recent call last):
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command", line 3, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: import pscheduler
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command", line 4, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command", line 14, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command", line 7, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command"
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function test_boot() line 10 at assignment
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT test_boot()"
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function warm_boot() line 4 at PERFORM
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT warm_boot()"
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function cold_boot() line 4 at PERFORM
Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.
From: <>
on behalf of Pennington, Mike <>
Date: Tuesday, September 5, 2023 at 9:01 AM
To: <>
Subject: RE: [perfsonar-user] pscheduler service status (Not running) on Dashboard
CAUTION: This email originated from outside of NREL. Do not
click links or open attachments unless you recognize the sender and know the content is safe.
I have this problem as well, but only on my Redhat 9.2 perfsonar boxes, what version OS are you running on this one?
From: <> On
Behalf Of Syed Asif Shah
Sent: Tuesday, September 5, 2023 10:56 AM
To: Wilson Cheung <>
Subject: [perfsonar-user] pscheduler service status (Not running) on Dashboard
*Message sent from a system outside of UConn.*
I found an strange status of pscheduler service on Toolkit dashboard, it is showing that service is "Not running" but when I checked the services on server it is showing all the services are running including
pscheduler. Any idea that where is the problem? I have attached screenshot.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient,
be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return
mail.
--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user
|
