perfSONAR User Q&A and Other Discussion

[Onno Zweers <>]

Hi all,

 

On our perfsonar system, we have an automatic yum update running every Tuesday. This is to make sure that any security issues are patched very soon. We consider this good practice since the box is open to the world.

 

Today our monitoring warned me that /tmp on our perfsonar node was full. After investigation, I discovered the upgraded packages. I had missed the announcement, apologies for that (too busy to read emails haha).

 

So now we have a system that says “Error loading test listing; measurement archive unreachable:

https://perfsonar-bandwidth.grid.surfsara.nl/esmond/perfsonar/archive/”

 

I tried some simple troubleshooting steps, like this:

 

[root@perfsonar /var/log/esmond]# pscheduler troubleshoot

Performing basic troubleshooting of perfsonar.grid.surfsara.nl.

 

perfsonar.grid.surfsara.nl:

 

  Measuring MTU... 65535 (Local)

  Looking for pScheduler... OK.

  Fetching API level... 5

  Checking clock... OK.

  Exercising API... Archivers... Contexts... Tests... Tools... OK.

  Fetching service status... OK.

  Checking services... Ticker... Scheduler... Runner... Archiver... OK.

  Checking limits... Failed.

 

Limit processor is not initialized: Limit 'idleex-default' has unsupported type 'test'

 

And I found a reference to perfsonar-pscheduler-limit-checker-for-5.0 which I tried:

 

[root@perfsonar /tmp/check/perfsonar-pscheduler-limit-checker-for-5.0-c8d6f4d]# /usr/bin/env python3 check-limits

NOTE: This limit configration could not be fully validated on a perfSONAR 5.0.0  system.  This program will still attempt to find problems related to the removal of

the 'test' limit.

 

ACTION REQUIRED.

 

This limit system configuration contains limits that are deprecated in

the 4.x family and will not be supported in release 5.0.

 

Limit 'idleex-default':

 

This limit has been removed in 5.0.  The idleex test is now disallowed for untrusted hosts by the 'allowed-tests' limit.

 

 

Limit 'throughput-default-time':

 

This limit has been rewritten to be compatible with perfSONAR 5.0.  If yours has not been customized, it can be replaced in its entirety with the following:

    

    {

        "name": "throughput-default-time",

        "description": "Throughput tests of reasonable duration",

        "type": "jq",

        "data": {

            "script": [

                "import \"pscheduler/iso8601\" as iso;",

                "if .test.type != \"throughput\"",

                "then true  # Don't care.",

                "else",

          "  if .test.spec.duration != null",

          "  and iso::duration_as_seconds(.test.spec.duration) > 60",

                "  then \"Duration for throughput must be 60 seconds or less.\"",

                "  else true end",

                "end"

            ]

        }

    }

    

Note that any of the modifications described above will be fully-compatible with 4.4.x  Getting a revised version tested and into production prior to upgrading to

5.0 is strongly-recommended.

 

It says “replace” but I can’t seem to find which file or config I need to replace.

 

I also noptice that opensearch was installed together with perfsonar 5.0. But it failes to start.

 

[root@perfsonar ~]# systemctl status opensearch.service

● opensearch.service - OpenSearch

   Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Tue 2023-04-18 16:31:43 CEST; 4s ago

     Docs: https://opensearch.org/

  Process: 88115 ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet (code=exited, status=1/FAILURE)

 Main PID: 88115 (code=exited, status=1/FAILURE)

 

Apr 18 16:31:40 perfsonar.grid.surfsara.nl systemd-entrypoint[88115]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)

Apr 18 16:31:40 perfsonar.grid.surfsara.nl systemd-entrypoint[88115]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)

Apr 18 16:31:40 perfsonar.grid.surfsara.nl systemd-entrypoint[88115]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)

Apr 18 16:31:40 perfsonar.grid.surfsara.nl systemd-entrypoint[88115]: at org.opensearch.cli.Command.main(Command.java:101)

Apr 18 16:31:40 perfsonar.grid.surfsara.nl systemd-entrypoint[88115]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)

Apr 18 16:31:40 perfsonar.grid.surfsara.nl systemd-entrypoint[88115]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

Apr 18 16:31:43 perfsonar.grid.surfsara.nl systemd[1]: opensearch.service: main process exited, code=exited, status=1/FAILURE

Apr 18 16:31:43 perfsonar.grid.surfsara.nl systemd[1]: Failed to start OpenSearch.

Apr 18 16:31:43 perfsonar.grid.surfsara.nl systemd[1]: Unit opensearch.service entered failed state.

Apr 18 16:31:43 perfsonar.grid.surfsara.nl systemd[1]: opensearch.service failed.

 

 

I’m not sure what to do now. Any suggestions?

 

Cheers,

Onno