perfSONAR User Q&A and Other Discussion

["Fedorka, Shayne" <>]

Thanks for looking into this Ivan. Something must have gone wrong with my upgrade from Ubuntu 18.04 to 20.04 and/or perfsonar v4 to v5.

 

I just did a clean install of 20.04 on the server followed by perfsonar v5 and all services are running without issue.

 

From: Garnizov, Ivan <>
Date: Friday, September 8, 2023 at 9:31 AM
To: Fedorka, Shayne <>, perfsonar-user () <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi all,

 

I am not sure about the problem and the observation of Shayne, but…

I have completed a fresh install of the pS Testpoint 5.0.4 on Ubuntu 20.04 and have seen no issues.

 

 

@r72:~$ systemctl status pscheduler-ticker

в—Џ pscheduler-ticker.service - pScheduler server - ticker

     Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)

     Active: active (running) since Fri 2023-09-08 15:18:34 UTC; 7min ago

   Main PID: 13392 (python3)

      Tasks: 2 (limit: 7007)

     Memory: 21.0M

     CGroup: /system.slice/pscheduler-ticker.service

             в””в”Ђ13392 python3 /usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn --pid-file /var/run/pscheduler-server/ticker/pid

 

Sep 08 15:18:34 r72.rrze systemd[1]: Starting pScheduler server - ticker...

Sep 08 15:18:34 r72.rrze systemd[1]: Started pScheduler server - ticker.

Sep 08 15:18:34 r72.rrze python3[13392]: ticker INFO     Started

@r72:~$ dpkg -l | grep pscheduler-server

ii  pscheduler-server                     5.0.4-1                           all          pScheduler server components

 

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP9T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

From: Fedorka, Shayne [mailto:]
Sent: Wednesday, September 6, 2023 5:25 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard

 

python3 -m pycurl

/usr/bin/python3: No code object available for pycurl

 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:24 AM
To: Fedorka, Shayne <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

What is the output of:

 

python3 -m pycurl 

 

 

From: Fedorka, Shayne []
Sent: Wednesday, September 6, 2023 5:17 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard

 

I just confirmed, that package is installed. Any other ideas?

 

apt list --installed | grep pycurl

 

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

 

python3-pycurl/focal,now 7.43.0.2-1ubuntu5 amd64 [installed]

 

 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 9:12 AM
To: Fedorka, Shayne <>
Cc: perfsonar-user () <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Fedorka,

 

The real problem is actually in this error message:

WARNING  Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'

 

You are most likely somehow missing this package

python3-pycurl

 

All credits to Mark Feit for the information comes from him.

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP9T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

 

From: Fedorka, Shayne []
Sent: Wednesday, September 6, 2023 5:05 PM
To: Garnizov, Ivan (RRZE) <>
Subject: Re: pscheduler service status (Not running) on Dashboard

 

Hi Ivan,

 

The output of the requested command does not return anything.

 

Yes, the one command did get cut, here is the full output:

Process: 576009 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pscheduler-server/ticker/options; chown pscheduler:pscheduler /var

 

From: Garnizov, Ivan <>
Date: Wednesday, September 6, 2023 at 6:51 AM
To: Fedorka, Shayne <>
Subject: RE: pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Dear Shayne,

 

Indeed this is a strange problem. I am myself running a test pS instance of Ubuntu 20.04.

Please share the output of this command:

sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf

 

Is there some cutting of text from the diagnostic window?

I am not sure this command from your output makes sense:

echo OPTIONS=$opts > /var/run/pschedule

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP9T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From: [] On Behalf Of "Fedorka, Shayne"
Sent: Tuesday, September 5, 2023 10:00 PM
To:
Subject: Re: [perfsonar-user] pscheduler service status (Not running) on Dashboard

 

I also have this issue after upgrading from perfsonar version 4 to 5 on Ubuntu 20.04. I found the pscheduler-ticker service is failing but I’m not sure why.

 

sudo systemctl status pscheduler-ticker.service

â— pscheduler-ticker.service - pScheduler server - ticker

     Loaded: loaded (/lib/systemd/system/pscheduler-ticker.service; enabled; vendor preset: enabled)

     Active: activating (auto-restart) (Result: exit-code) since Tue 2023-09-05 13:26:54 MDT; 2s ago

    Process: 3284 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

    Process: 3285 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

    Process: 3286 ExecStartPre=/bin/chmod 755 /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

    Process: 3287 ExecStartPre=/bin/mkdir -p /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)

    Process: 3288 ExecStartPre=/bin/chmod 700 /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)

    Process: 3289 ExecStartPre=/bin/chown pscheduler:pscheduler /var/run/pscheduler-server/ticker/tmp (code=exited, status=0/SUCCESS)

    Process: 3290 ExecStartPre=/bin/sh -c if [ -r /etc/pscheduler/daemons/ticker.conf ]; then opts=$(sed -e 's/#.*$//' /etc/pscheduler/daemons/ticker.conf); echo OPTIONS=$opts > /var/run/pschedule>

    Process: 3293 ExecStart=/usr/lib/pscheduler/daemons/ticker --dsn @/etc/pscheduler/database/database-dsn $OPTIONS --pid-file /var/run/pscheduler-server/ticker/pid (code=exited, status=1/FAILURE)

    Process: 3300 ExecStopPost=/bin/rm -rf /var/run/pscheduler-server/ticker (code=exited, status=0/SUCCESS)

   Main PID: 3293 (code=exited, status=1/FAILURE)

 

Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function warm_boot() line 4 at PERFORM

Sep 05 13:26:58 esif-perf05-10g ticker[3323]: SQL statement "SELECT warm_boot()"

Sep 05 13:26:58 esif-perf05-10g ticker[3323]: PL/pgSQL function cold_boot() line 4 at PERFORM

Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE

Sep 05 13:26:58 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.

Sep 05 13:27:01 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Scheduled restart job, restart counter is at 34.

Sep 05 13:27:01 esif-perf05-10g systemd[1]: Stopped pScheduler server - ticker.

Sep 05 13:27:01 esif-perf05-10g systemd[1]: Starting pScheduler server - ticker...

Sep 05 13:27:01 esif-perf05-10g systemd[1]: Started pScheduler server - ticker.

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker INFO     Started

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  Queue maintainer got exception ModuleNotFoundError: No module named 'pycurl'

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  CONTEXT:  Traceback (most recent call last):

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 3, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING      import pscheduler

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 4, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 14, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING    PL/Python function "http_delete", line 7, in <module>

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/Python function "http_delete"

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/pgSQL function http_queue_process(bigint) line 19 at assignment

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  SQL statement "SELECT http_queue_process(id) FROM http_queue

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING      WHERE next_attempt < now() OR attempts = 0"

Sep 05 13:27:02 esif-perf05-10g python3[3349]: ticker WARNING  PL/pgSQL function http_queue_process_all() line 3 at PERFORM

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: Traceback (most recent call last):

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   File "/usr/lib/pscheduler/daemons/ticker", line 172, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     main_program()

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   File "/usr/lib/pscheduler/daemons/ticker", line 132, in main_program

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     cursor.execute("SELECT cold_boot()")

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: psycopg2.errors.ExternalRoutineException: ModuleNotFoundError: No module named 'pycurl'

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: CONTEXT:  Traceback (most recent call last):

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 3, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:     import pscheduler

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 4, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 14, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]:   PL/Python function "pscheduler_command", line 7, in <module>

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/Python function "pscheduler_command"

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function test_boot() line 10 at assignment

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT test_boot()"

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function warm_boot() line 4 at PERFORM

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: SQL statement "SELECT warm_boot()"

Sep 05 13:27:02 esif-perf05-10g ticker[3349]: PL/pgSQL function cold_boot() line 4 at PERFORM

Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Main process exited, code=exited, status=1/FAILURE

Sep 05 13:27:02 esif-perf05-10g systemd[1]: pscheduler-ticker.service: Failed with result 'exit-code'.

 

 

From: <> on behalf of Pennington, Mike <>
Date: Tuesday, September 5, 2023 at 9:01 AM
To: <>
Subject: RE: [perfsonar-user] pscheduler service status (Not running) on Dashboard

CAUTION: This email originated from outside of NREL. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I have this problem as well, but only on my Redhat 9.2 perfsonar boxes, what version OS are you running on this one?

 

Mike Pennington CEN | Network Engineer Hartford CT | 06105-3702 o 860 622 4566 Visit our Website

 

 

From: <> On Behalf Of Syed Asif Shah
Sent: Tuesday, September 5, 2023 10:56 AM
To: Wilson Cheung <>
Subject: [perfsonar-user] pscheduler service status (Not running) on Dashboard

 

*Message sent from a system outside of UConn.*

 

Hi,

   I found an strange status of pscheduler service on Toolkit dashboard, it is showing that service is "Not running" but when I checked the services on server it is showing all the services are running including pscheduler. Any idea that where is the problem? I have attached screenshot.

 

 

 

Thanks