perfSONAR User Q&A and Other Discussion

[Tao Zhang <>]

Hi Ivan,

All our perfSONAR boxes are pointing to a single remote configuration file.  I checked our configuration file and did not find anything suspicious.

Here are the outputs of some psconfig commands.

psconfig remote list

=== pScheduler Agent ===

[

   {

      "url" : "https://lat-otwa3pfs1.network.canarie.ca/canarieTest.json"

   }

]

psconfig validate --deep https://lat-otwa3pfs1.network.canarie.ca/canarieTest.json

Loading template ...... OK

Validating JSON schema ...... OK

Verifying object references ...... OK

pScheduler Validation (Deep) ...... OK

pSConfig JSON is valid

The issue was caused by a weird timestamp, 

https://lat-clgr2pfs1.network.canarie.ca/pscheduler/tasks?expanded&detail&pretty (look at the "added" timestamps and you'll see the strange format).

[
    {
        "_key": null,
        "archives": [
            {
                "archiver": "esmond",
                "data": {
                    "_auth-token": null,
                    "measurement-agent": "bdw-clgr2pfs1.network.canarie.ca",
                    "url": "https://bdw-clgr2pfs1.network.canarie.ca/esmond/perfsonar/archive/"
                }
            }
        ],
        "detail": {
            "added": "2022-01-25T21:02:33-07",
            "anytime": false,
            "cli": [
                "--source",
                "bdw-hlfx1pfs1.network.canarie.ca",
                "--source-node",
                "bdw-hlfx1pfs1.network.canarie.ca",
                "--dest",
                "bdw-clgr2pfs1.network.canarie.ca",
                "--dest-node",
                "bdw-clgr2pfs1.network.canarie.ca",
                "--duration",
                "PT30S",
                "--window-size",
                "67108864",
                "--omit",
                "PT5S"
            ],

          ...............

We are not sure where this timestamp came from. Any thought on this issue would be appreciated.

Thanks

Tao

---

From: <> on behalf of Garnizov, Ivan <>
Sent: Friday, January 21, 2022 7:54 AM
To: perfsonar-user () <>
Subject: RE: [perfsonar-user] psconfig service not running

 

---

External This email originated from outside the organization. Use caution when following links as they could open malicious web sites.

---

Hi Tao,

 

The most likely reason for your failure is bad configuration file. I presume you have had a fully functioning system before this date.

The most likely configuration error comes with manual changes in a mesh configuration file.

You are not sharing anything about missing measurements / results, so I presume the system works now on an “auto-pilot”.

 

 

Since this is a Toolkit deployment, the mesh configuration could be changed centrally (remotely) or locally through the GUI.

It could be the case, that this system is part of a big mesh managed by another entity.

 

Please share some more info on how the measurement requests are being setup for this system.

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

 

From: [] On Behalf Of Tao Zhang
Sent: Thursday, January 20, 2022 6:06 PM
To:
Subject: [perfsonar-user] psconfig service not running

 

Hi team,

Started from 2021/12/21, the psconfig service on our perfSONAR boxes has been showing 'not running'. I have restarted psconfig-pscheduler-agent service several times, service will go up for a couple of minutes before shutting itself down again.  

 

service psconfig-pscheduler-agent status

Redirecting to /bin/systemctl status psconfig-pscheduler-agent.service

● psconfig-pscheduler-agent.service - pSConfig PScheduler Agent

   Loaded: loaded (/usr/lib/systemd/system/psconfig-pscheduler-agent.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Thu 2022-01-20 11:39:44 EST; 16min ago

  Process: 1917 ExecStart=/usr/lib/perfsonar/bin/psconfig_pscheduler_agent --config=/etc/perfsonar/psconfig/pscheduler-agent.json --logger=/etc/perfsonar/psconfig/pscheduler-agent-logger.conf --pidfile=/var/run/psconfig-pscheduler-agent.pid --user=perfsonar --group=perfsonar (code=exited, status=0/SUCCESS)

 Main PID: 1928 (code=exited, status=25)

Here is the ERROR message in psconfig-pscheduler-agent.log

 

2021/12/22 02:00:07 INFO pid=17854 prog=main:: line=166 msg=Starting pSConfig pScheduler agent

2021/12/22 02:00:07 INFO pid=17854 prog=main:: line=176 guid=CC32A336-62F4-11EC-AB30-A66E5A4D48ED msg=Running agent...

2021/12/22 02:01:21 ERROR pid=17854 prog=main::__ANON__ line=131 guid=CC32A336-62F4-11EC-AB30-A66E5A4D48ED msg=Died: Invalid date format: 2021-12-21T12:34:09-03:-30 at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/Client/PScheduler/TaskManager.pm line 565.

        perfSONAR_PS::Client::PScheduler::TaskManager::_iso_to_ts('perfSONAR_PS::Client::PScheduler::TaskManager=HASH(0x6537df0)', '2021-12-21T12:34:09-03:-30') called at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/Client/PScheduler/TaskManager.pm line 395

        perfSONAR_PS::Client::PScheduler::TaskManager::_evaluate_task('perfSONAR_PS::Client::PScheduler::TaskManager=HASH(0x6537df0)', 'HASH(0x67e7fe0)', 1, undef) called at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/Client/PScheduler/TaskManager.pm line 368

        perfSONAR_PS::Client::PScheduler::TaskManager::_need_new_task('perfSONAR_PS::Client::PScheduler::TaskManager=HASH(0x6537df0)', 'perfSONAR_PS::Client::PScheduler::Task=HASH(0x66bd548)') called at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/Client/PScheduler/TaskManager.pm line 204

        perfSONAR_PS::Client::PScheduler::TaskManager::add_task('perfSONAR_PS::Client::PScheduler::TaskManager=HASH(0x6537df0)', 'task', 'perfSONAR_PS::Client::PScheduler::Task=HASH(0x66bd548)') called at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/PSConfig/PScheduler/Agent.pm line 210

        perfSONAR_PS::PSConfig::PScheduler::Agent::_run_handle_psconfig('perfSONAR_PS::PSConfig::PScheduler::Agent=HASH(0x2755800)', 'perfSONAR_PS::Client::PSConfig::Config=HASH(0x6b852b8)', 'perfSONAR_PS::PSConfig::PScheduler::Config=HASH(0x64deb00)', 'perfSONAR_PS::PSConfig::Remote=HASH(0x6b8c188)') called at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/PSConfig/BaseAgent.pm line 364

        perfSONAR_PS::PSConfig::BaseAgent::run('perfSONAR_PS::PSConfig::PScheduler::Agent=HASH(0x2755800)') called at /usr/lib/perfsonar/bin/psconfig_pscheduler_agent line 177

 

 

From the history email, I saw Bruce Curtis from NDSU had similar issue. Not sure if this issue has been resolved. 

 

Thanks,

Tao