Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard


Chronological Thread 
  • From: Mark Chouteau <>
  • To: "Horstmann, Ben" <>
  • Cc: Wilson Cheung <>, "Garnizov, Ivan" <>, "perfsonar-user ()" <>, "Fedorka, Shayne" <>, Robert Lageano <>
  • Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
  • Date: Mon, 11 Sep 2023 15:17:37 -0400

I see this issue also, Building out new nodes in the last 3 weeks:
New CENT install is fine.
New installs using Ubuntu 20.04 on bare metal and VM both have the issue of not showing test results. 

I have experienced the “Not Running” issue once.

Mark J. Chouteau
Sr. Network Engineer
Miami University of Ohio



On Sep 11, 2023, at 2:41 PM, Horstmann, Ben <> wrote:

I am experiencing the same behavior as Ivan and Wilson.  Clean installs of Ubuntu and Alma experience this.  Have not tried CENT.
 
We are using the traditional repo, not the early release.  Are the fixes perhaps included in EL and not yet in prod?  Thanks!
 
Best,
Ben
 
Ben Horstmann
Assistant Director, Network Operations Center
Washington University in St. Louis
4480 Clayton Ave, St. Louis, MO 63110
Find me on MS Teams here
314-308-5839 (c)
 
 
 
 
 
From: <> on behalf of Wilson Cheung <>
Reply-To: Wilson Cheung <>
Date: Monday, September 11, 2023 at 1:31 PM
To: "Garnizov, Ivan" <>, "perfsonar-user ()" <>, "Fedorka, Shayne" <>, Robert Lageano <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
 
perfSONAR team,
 
I'll chime in and say that I'm also experiencing the same problems with fresh installs of Ubuntu Server 20.04.6 with latest OS updates plus latest perfSONAR 5.0.4 Toolkit.  I did the "apt install perfsonar-toolkit" option as described at "https://docs.perfsonar.net/install_debian.html".
 
Nothing ever shows up in the "Test Results" section of the Toolkit webpage.  Have tried on multiple Ubuntu installs on both physical and virtual machine servers with a mix of 1GigE/10GigE NICs.  I'm able to login via the toolkit webpages and add more  tests to run more frequently (e.g. every 15-minutes) to see if anything would show up.
 
I did a separate full pS 5.0.4 "toolkit" install on a fresh install of CentOS 7.9-2009 plus latest OS updates to compare and results do appear under the "Test Results" page, so this problem may be Ubuntu specific.  Haven't tried other Debian-based distros yet.
 
On the Toolkit webpage of the Ubuntu servers, it says "pscheduler" is "Running" in green.  If I leave it in this state and don't reboot the servers for a few days, the pscheduler status tends to eventually changes to "Not Running" in red.  If I manually restart all of the pscheduler services, it goes back to "Running".  Still no "Test Results" appear locally.
 
On the Ubuntu servers, I also see the same errors in the "pscheduler.log "that Ivan listed about "Failed to archive" and "Failed to put result: 503" and "503 Service Unavailable".
 
I also see other errors in "logstash-plain.log" and "opensearch.log" about Java.  A recent post talked about making sure your servers are not under-resourced and allocating more memory resources to the JVM used by opensearch/logstash.
 
Before I dig further down into the various pS internals, are there other specific logs or others I should focus on checking?  I'm not sure if there's something pS configuration specific I missed.
 
Thanks!
 
-- 
Wilson Cheung
 

From:  <> on behalf of Robert Lageano <>
Sent: Sunday, September 10, 2023 11:57 PM
To: Garnizov, Ivan <>; perfsonar-user () <>; Fedorka, Shayne <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
 
I have a fresh install of Ubuntu Server 20.04 and Perfsonar Toolkit but it refuses to show the test results. Can someone help me with the following issue:
 
● pscheduler-archiver.service - pScheduler server - archiver
     Loaded: loaded (/lib/systemd/system/pscheduler-archiver.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-09-11 01:08:16 UTC; 5h 47min ago
    Process: 61558 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
    Process: 61563 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
    Process: 61567 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)
    Process: 61569 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
    Process: 61571 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
    Process: 61574 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)
    Process: 61577 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/archiver.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/archiver.conf); echo OPTIONS=$opts > /var/run/pscheduler-server/archiver/options; chown pscheduler:pscheduler /var/run/pscheduler-server/archiver/options; fi (code=exited, status=0/SUCCESS)
   Main PID: 61585 (python3)
      Tasks: 2 (limit: 202842)
     Memory: 58.3M
     CGroup: /system.slice/pscheduler-archiver.service
             └─61585 python3 /usr/lib/pscheduler/daemons/archiver --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/archiver/pid
 
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  94: Failed to archive https://perfsonar/pscheduler/tasks/42b16acb-b0a2-4f2a-8363-017df2436d7b/runs/47bb3ebe-78de-411f-a3ce-6d53678aaf51 to http: Failed to put result: 503: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <html><head>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <title>503 Service Unavailable</title>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  </head><body>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <h1>Service Unavailable</h1>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <p>The server is temporarily unable to service your
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  request due to maintenance downtime or capacity
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  problems. Please try again later.</p>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  </body></html>
Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  94: Gave up archiving https://perfsonar/pscheduler/tasks/42b16acb-b0a2-4f2a-8363-017df2436d7b/runs/47bb3ebe-78de-411f-a3ce-6d53678aaf51 to http
 
Cheers,


Rob
-----------------------------------------------------------------------------
Robert Lageano
Senior IT Network Security Specialist
National Computational Infrastructure (NCI)
The Australian National University
143 Ward Road
Acton, ACT, 2601
M: +61 452 535 678


Want the latest news from NCI? nci.org.au

Find out more about NCI: YouTube | Facebook | Twitter LinkedIn Podcast



<image001.png>

 

 




From:  <> on behalf of "Fedorka, Shayne" <>
Sent: Saturday, September 9, 2023 7:12 AM
To: Garnizov, Ivan <>; perfsonar-user () <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
 
Thanks for looking into this Ivan. Something must have gone wrong with my upgrade from Ubuntu 18.04 to 20.04 and/or perfsonar v4 to v5.
 
I just did a clean install of 20.04 on the server followed by perfsonar v5 and all services are running without issue.
 

From: Garnizov, Ivan <>
Date: Friday, September 8, 2023 at 9:31 AM
To: Fedorka, Shayne <>, perfsonar-user () <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi all,
 
I am not sure about the problem and the observation of Shayne, but…
I have completed a fresh install of the pS Testpoint 5.0.4 on Ubuntu 20.04 and have seen no issues.
 
 
@r72:~$ systemctl status pscheduler-ticker
в—Џ pscheduler-ticker.service - pScheduler server - ticker
     Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-09-08 15:18:34 UTC; 7min ago
   Main PID: 13392 (python3)
      Tasks: 2 (limit: 7007)
     Memory: 21.0M
     CGroup: /system.slice/pscheduler-ticker.service
             в””в”Ђ13392 python3 /usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/ticker/pid
 
Sep 08 15:18:34 r72.rrze systemd[1]: Starting pScheduler server - ticker...
Sep 08 15:18:34 r72.rrze systemd[1]: Started pScheduler server - ticker.
Sep 08 15:18:34 r72.rrze python3[13392]: ticker INFO     Started
@r72:~$ dpkg -l | grep pscheduler-server
ii  pscheduler-server                     5.0.4-1                           all          pScheduler server components
 
 
 
Regards,
Ivan Garnizov
 
GEANT WP6T3: pS development team
GEANT WP9T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
 
 
From: Fedorka, Shayne [] 
Sent: Wednesday, September 6, 2023 5:25 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard
 
python3 -m pycurl
/usr/bin/python3: No code object available for pycurl
 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:24 AM
To: Fedorka, Shayne <
>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

What is the output of:
 
python3 -m pycurl 
 
 
From: Fedorka, Shayne [] 
Sent: Wednesday, September 6, 2023 5:17 PM
To: Garnizov, Ivan (RRZE) <
>
Subject: Re: pscheduler service status (Not running) on Dashboard
 
I just confirmed, that package is installed. Any other ideas?
 
apt list --installed | grep pycurl
 
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
 
python3-pycurl/focal,now 7.43.0.2-1ubuntu5 amd64 [installed]
 
 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:12 AM
To: Fedorka, Shayne <
>
Cc: perfsonar-user (
) <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Fedorka,
 
The real problem is actually in this error message:
WARNING  Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'
 
You are most likely somehow missing this package
python3-pycurl
 
All credits to Mark Feit for the information comes from him.
 
Regards,
Ivan Garnizov
 
GEANT WP6T3: pS development team
GEANT WP9T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
 
 
 
 
From: Fedorka, Shayne [] 
Sent: Wednesday, September 6, 2023 5:05 PM
To: Garnizov, Ivan (RRZE) <
>
Subject: Re: pscheduler service status (Not running) on Dashboard
 
Hi Ivan,
 
The output of the requested command does not return anything.
 
Yes, the one command did get cut, here is the full output:
Process: 576009 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pscheduler-server/ticker/options; chown pscheduler:pscheduler /var
 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 6:51 AM
To: Fedorka, Shayne <
>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Dear Shayne,
 
Indeed this is a strange problem. I am myself running a test pS instance of Ubuntu 20.04.
Please share the output of this command:
sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf
 
Is there some cutting of text from the diagnostic window?
I am not sure this command from your output makes sense:
echo OPTIONS=$opts > /var/run/pschedule
 
 
Regards,
Ivan Garnizov
 
GEANT WP6T3: pS development team
GEANT WP9T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
 
 
 
From:  [] On Behalf Of "Fedorka, Shayne"
Sent: Tuesday, September 5, 2023 10:00 PM
To: 

Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard
 
I also have this issue after upgrading from perfsonar version 4 to 5 on Ubuntu 20.04. I found the pscheduler-ticker service is failing but I’m not sure why.
 
sudo systemctl status pscheduler-ticker.service
â— pscheduler-ticker.service - pScheduler server - ticker
     Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2023-09-05 13:26:54 MDT; 2s ago
    Process: 3284 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
    Process: 3285 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
    Process: 3286 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
    Process: 3287 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
    Process: 3288 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
    Process: 3289 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)
    Process: 3290 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pschedule>
    Process: 3293 ExecStart=/usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn $OPTIONS --pid-file /var/run/pscheduler-server/ticker/pid (code=exited, status=1/FAILURE)
    Process: 3300 ExecStopPost=/bin/rm -rf /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)
   Main PID: 3293 (code=exited, status=1/FAILURE)
 
Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function warm_boot() line 4 at PERFORM
Sep 05 13:26:58 esif-perf05-10g ticker[3323]: SQL statement "SELECT warm_boot()"
Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function cold_boot() line 4 at PERFORM
Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.
Sep 05 13:27:01 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Scheduled restart job, restart counter is at 34.
Sep 05 13:27:01 esif-perf05-10g systemd[1]: Stopped pScheduler server - ticker.
Sep 05 13:27:01 esif-perf05-10g systemd[1]: Starting pScheduler server - ticker...
Sep 05 13:27:01 esif-perf05-10g systemd[1]: Started pScheduler server - ticker.
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker INFO     Started
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  CONTEXT:  Traceback (most recent call last):
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 3, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING      import pscheduler
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 4, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 14, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 7, in <module>
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/Python function "http_delete"
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/pgSQL function http_queue_process(bigint) line 19 at assignment
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  SQL statement "SELECT http_queue_process(id) FROM http_queue
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING      WHERE next_attempt < now() OR attempts = 0"
Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/pgSQL function http_queue_process_all() line 3 at PERFORM
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: Traceback (most recent call last):
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   File "/usr/lib/pscheduler/daemons/ticker", line 172, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     main_program()
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   File "/usr/lib/pscheduler/daemons/ticker", line 132, in main_program
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     cursor.execute("SELECT cold_boot()")
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: psycopg2.errors.ExternalRoutineException: ModuleNotFoundError: No module named 'pycurl'
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: CONTEXT:  Traceback (most recent call last):
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 3, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     import pscheduler
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 4, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 14, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 7, in <module>
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command"
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function test_boot() line 10 at assignment
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT test_boot()"
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function warm_boot() line 4 at PERFORM
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT warm_boot()"
Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function cold_boot() line 4 at PERFORM
Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.
 
 

From:  <> on behalf of Pennington, Mike <>
Date: Tuesday, September 5, 2023 at 9:01 AM
To: 
 <>
Subject: RE: [perfsonar-user] pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I have this problem as well, but only on my Redhat 9.2 perfsonar boxes, what version OS are you running on this one?
 
 
 
  <image004.png>  <image005.png>  <image006.png>
Mike Pennington
CEN | Network Engineer
Hartford CT | 06105-3702
o 860 622 4566
 
 
From:  <> On Behalf Of Syed Asif Shah
Sent: Tuesday, September 5, 2023 10:56 AM
To: Wilson Cheung <
>
Subject: [perfsonar-user] pscheduler service status (Not running) on Dashboard
 
*Message sent from a system outside of UConn.*
 
Hi,
   I found an strange status of pscheduler service on Toolkit dashboard, it is showing that service is "Not running" but when I checked the services on server it is showing all the services are running including pscheduler. Any idea that where is the problem? I have attached screenshot.
 
 
 
Thanks
 
 
 
 
 

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user




Archive powered by MHonArc 2.6.24.

Top of Page