perfSONAR User Q&A and Other Discussion

[Mark Chouteau <>]

I see this issue also, Building out new nodes in the last 3 weeks:

New CENT install is fine.

New installs using Ubuntu 20.04 on bare metal and VM both have the issue of not showing test results. 

I have experienced the “Not Running” issue once.

Mark J. Chouteau

Sr. Network Engineer

Miami University of Ohio

On Sep 11, 2023, at 2:41 PM, Horstmann, Ben <> wrote:

I am experiencing the same behavior as Ivan and Wilson.  Clean installs of Ubuntu and Alma experience this.  Have not tried CENT.

 

We are using the traditional repo, not the early release.  Are the fixes perhaps included in EL and not yet in prod?  Thanks!

 

Best,

Ben

 

Ben Horstmann

Assistant Director, Network Operations Center

Washington University in St. Louis

4480 Clayton Ave, St. Louis, MO 63110

Find me on MS Teams here

314-308-5839 (c)

 

 

 

 

 

From: <> on behalf of Wilson Cheung <>
Reply-To: Wilson Cheung <>
Date: Monday, September 11, 2023 at 1:31 PM
To: "Garnizov, Ivan" <>, "perfsonar-user ()" <>, "Fedorka, Shayne" <>, Robert Lageano <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard

 

perfSONAR team,

 

I'll chime in and say that I'm also experiencing the same problems with fresh installs of Ubuntu Server 20.04.6 with latest OS updates plus latest perfSONAR 5.0.4 Toolkit.  I did the "apt install perfsonar-toolkit" option as described at "https://docs.perfsonar.net/install_debian.html".

 

Nothing ever shows up in the "Test Results" section of the Toolkit webpage.  Have tried on multiple Ubuntu installs on both physical and virtual machine servers with a mix of 1GigE/10GigE NICs.  I'm able to login via the toolkit webpages and add more  tests to run more frequently (e.g. every 15-minutes) to see if anything would show up.

 

I did a separate full pS 5.0.4 "toolkit" install on a fresh install of CentOS 7.9-2009 plus latest OS updates to compare and results do appear under the "Test Results" page, so this problem may be Ubuntu specific.  Haven't tried other Debian-based distros yet.

 

On the Toolkit webpage of the Ubuntu servers, it says "pscheduler" is "Running" in green.  If I leave it in this state and don't reboot the servers for a few days, the pscheduler status tends to eventually changes to "Not Running" in red.  If I manually restart all of the pscheduler services, it goes back to "Running".  Still no "Test Results" appear locally.

 

On the Ubuntu servers, I also see the same errors in the "pscheduler.log "that Ivan listed about "Failed to archive" and "Failed to put result: 503" and "503 Service Unavailable".

 

I also see other errors in "logstash-plain.log" and "opensearch.log" about Java.  A recent post talked about making sure your servers are not under-resourced and allocating more memory resources to the JVM used by opensearch/logstash.

 

Before I dig further down into the various pS internals, are there other specific logs or others I should focus on checking?  I'm not sure if there's something pS configuration specific I missed.

 

Thanks!

 

-- 

Wilson Cheung

 

---

From:  <> on behalf of Robert Lageano <>
Sent: Sunday, September 10, 2023 11:57 PM
To: Garnizov, Ivan <>; perfsonar-user () <>; Fedorka, Shayne <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard

 

I have a fresh install of Ubuntu Server 20.04 and Perfsonar Toolkit but it refuses to show the test results. Can someone help me with the following issue:

 

● pscheduler-archiver.service - pScheduler server - archiver

     Loaded: loaded (/lib/systemd/system/pscheduler-archiver.service; enabled; vendor preset: enabled)

     Active: active (running) since Mon 2023-09-11 01:08:16 UTC; 5h 47min ago

    Process: 61558 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)

    Process: 61563 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)

    Process: 61567 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/archiver (code=exited, status=0/SUCCESS)

    Process: 61569 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)

    Process: 61571 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)

    Process: 61574 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/archiver/tmp (code=exited, status=0/SUCCESS)

    Process: 61577 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/archiver.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/archiver.conf); echo OPTIONS=$opts > /var/run/pscheduler-server/archiver/options; chown pscheduler:pscheduler /var/run/pscheduler-server/archiver/options; fi (code=exited, status=0/SUCCESS)

   Main PID: 61585 (python3)

      Tasks: 2 (limit: 202842)

     Memory: 58.3M

     CGroup: /system.slice/pscheduler-archiver.service

             └─61585 python3 /usr/lib/pscheduler/daemons/archiver --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/archiver/pid

 

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  94: Failed to archive https://perfsonar/pscheduler/tasks/42b16acb-b0a2-4f2a-8363-017df2436d7b/runs/47bb3ebe-78de-411f-a3ce-6d53678aaf51 to http: Failed to put result: 503: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <html><head>

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <title>503 Service Unavailable</title>

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  </head><body>

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <h1>Service Unavailable</h1>

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  <p>The server is temporarily unable to service your

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  request due to maintenance downtime or capacity

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  problems. Please try again later.</p>

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  </body></html>

Sep 11 06:52:41 perfsonar python3[273190]: archiver WARNING  94: Gave up archiving https://perfsonar/pscheduler/tasks/42b16acb-b0a2-4f2a-8363-017df2436d7b/runs/47bb3ebe-78de-411f-a3ce-6d53678aaf51 to http

 

Cheers,

Rob

-----------------------------------------------------------------------------

Robert Lageano
Senior IT Network Security Specialist
National Computational Infrastructure (NCI)
The Australian National University
143 Ward Road
Acton, ACT, 2601
M: +61 452 535 678

Want the latest news from NCI? nci.org.au

Find out more about NCI: YouTube | Facebook | Twitter | LinkedIn | Podcast

<image001.png>

 

 

---

From:  <> on behalf of "Fedorka, Shayne" <>
Sent: Saturday, September 9, 2023 7:12 AM
To: Garnizov, Ivan <>; perfsonar-user () <>
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard

 

Thanks for looking into this Ivan. Something must have gone wrong with my upgrade from Ubuntu 18.04 to 20.04 and/or perfsonar v4 to v5.

 

I just did a clean install of 20.04 on the server followed by perfsonar v5 and all services are running without issue.

 

From: Garnizov, Ivan <>
Date: Friday, September 8, 2023 at 9:31 AM
To: Fedorka, Shayne <>, perfsonar-user () <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi all,

 

I am not sure about the problem and the observation of Shayne, but…

I have completed a fresh install of the pS Testpoint 5.0.4 on Ubuntu 20.04 and have seen no issues.

 

 

@r72:~$ systemctl status pscheduler-ticker

в—Џ pscheduler-ticker.service - pScheduler server - ticker

     Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)

     Active: active (running) since Fri 2023-09-08 15:18:34 UTC; 7min ago

   Main PID: 13392 (python3)

      Tasks: 2 (limit: 7007)

     Memory: 21.0M

     CGroup: /system.slice/pscheduler-ticker.service

             в””в”Ђ13392 python3 /usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/ticker/pid

 

Sep 08 15:18:34 r72.rrze systemd[1]: Starting pScheduler server - ticker...

Sep 08 15:18:34 r72.rrze systemd[1]: Started pScheduler server - ticker.

Sep 08 15:18:34 r72.rrze python3[13392]: ticker INFO     Started

@r72:~$ dpkg -l | grep pscheduler-server

ii  pscheduler-server                     5.0.4-1                           all          pScheduler server components

 

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP9T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

From: Fedorka, Shayne [] 
Sent: Wednesday, September 6, 2023 5:25 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard

 

python3 -m pycurl

/usr/bin/python3: No code object available for pycurl

 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:24 AM
To: Fedorka, Shayne <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

What is the output of:

 

python3 -m pycurl 

 

 

From: Fedorka, Shayne [] 
Sent: Wednesday, September 6, 2023 5:17 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard

 

I just confirmed, that package is installed. Any other ideas?

 

apt list --installed | grep pycurl

 

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

 

python3-pycurl/focal,now 7.43.0.2-1ubuntu5 amd64 [installed]

 

 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:12 AM
To: Fedorka, Shayne <>
Cc: perfsonar-user () <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Fedorka,

 

The real problem is actually in this error message:

WARNING  Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'

 

You are most likely somehow missing this package

python3-pycurl

 

All credits to Mark Feit for the information comes from him.

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP9T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

 

From: Fedorka, Shayne [] 
Sent: Wednesday, September 6, 2023 5:05 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard

 

Hi Ivan,

 

The output of the requested command does not return anything.

 

Yes, the one command did get cut, here is the full output:

Process: 576009 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pscheduler-server/ticker/options; chown pscheduler:pscheduler /var

 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 6:51 AM
To: Fedorka, Shayne <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Dear Shayne,

 

Indeed this is a strange problem. I am myself running a test pS instance of Ubuntu 20.04.

Please share the output of this command:

sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf

 

Is there some cutting of text from the diagnostic window?

I am not sure this command from your output makes sense:

echo OPTIONS=$opts > /var/run/pschedule

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP9T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From:  [] On Behalf Of "Fedorka, Shayne"
Sent: Tuesday, September 5, 2023 10:00 PM
To: 
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard

 

I also have this issue after upgrading from perfsonar version 4 to 5 on Ubuntu 20.04. I found the pscheduler-ticker service is failing but I’m not sure why.

 

sudo systemctl status pscheduler-ticker.service

â— pscheduler-ticker.service - pScheduler server - ticker

     Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)

     Active: activating (auto-restart) (Result: exit-code) since Tue 2023-09-05 13:26:54 MDT; 2s ago

    Process: 3284 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

    Process: 3285 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

    Process: 3286 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

    Process: 3287 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)

    Process: 3288 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)

    Process: 3289 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)

    Process: 3290 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pschedule>

    Process: 3293 ExecStart=/usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn $OPTIONS --pid-file /var/run/pscheduler-server/ticker/pid (code=exited, status=1/FAILURE)

    Process: 3300 ExecStopPost=/bin/rm -rf /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

   Main PID: 3293 (code=exited, status=1/FAILURE)

 

Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function warm_boot() line 4 at PERFORM

Sep 05 13:26:58 esif-perf05-10g ticker[3323]: SQL statement "SELECT warm_boot()"

Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function cold_boot() line 4 at PERFORM

Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE

Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.

Sep 05 13:27:01 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Scheduled restart job, restart counter is at 34.

Sep 05 13:27:01 esif-perf05-10g systemd[1]: Stopped pScheduler server - ticker.

Sep 05 13:27:01 esif-perf05-10g systemd[1]: Starting pScheduler server - ticker...

Sep 05 13:27:01 esif-perf05-10g systemd[1]: Started pScheduler server - ticker.

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker INFO     Started

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  CONTEXT:  Traceback (most recent call last):

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 3, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING      import pscheduler

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 4, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 14, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 7, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/Python function "http_delete"

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/pgSQL function http_queue_process(bigint) line 19 at assignment

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  SQL statement "SELECT http_queue_process(id) FROM http_queue

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING      WHERE next_attempt < now() OR attempts = 0"

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/pgSQL function http_queue_process_all() line 3 at PERFORM

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: Traceback (most recent call last):

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   File "/usr/lib/pscheduler/daemons/ticker", line 172, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     main_program()

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   File "/usr/lib/pscheduler/daemons/ticker", line 132, in main_program

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     cursor.execute("SELECT cold_boot()")

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: psycopg2.errors.ExternalRoutineException: ModuleNotFoundError: No module named 'pycurl'

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: CONTEXT:  Traceback (most recent call last):

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 3, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     import pscheduler

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 4, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 14, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 7, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command"

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function test_boot() line 10 at assignment

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT test_boot()"

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function warm_boot() line 4 at PERFORM

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT warm_boot()"

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function cold_boot() line 4 at PERFORM

Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE

Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.

 

 

From:  <> on behalf of Pennington, Mike <>
Date: Tuesday, September 5, 2023 at 9:01 AM
To:  <>
Subject: RE: [perfsonar-user] pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I have this problem as well, but only on my Redhat 9.2 perfsonar boxes, what version OS are you running on this one?

 

<image002.png>     <image004.png>  <image005.png>  <image006.png>

Mike Pennington CEN | Network Engineer Hartford CT | 06105-3702 o 860 622 4566 Visit our Website

 

 

From:  <> On Behalf Of Syed Asif Shah
Sent: Tuesday, September 5, 2023 10:56 AM
To: Wilson Cheung <>
Subject: [perfsonar-user] pscheduler service status (Not running) on Dashboard

 

*Message sent from a system outside of UConn.*

 

Hi,

   I found an strange status of pscheduler service on Toolkit dashboard, it is showing that service is "Not running" but when I checked the services on server it is showing all the services are running including pscheduler. Any idea that where is the problem? I have attached screenshot.

 

 

 

Thanks

 

 

 

 

 

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user