perfSONAR User Q&A and Other Discussion

[Onno Zweers <>]

Hi all,

 

One of my problems has been fixed. Mary Hester @ Nikhef had a tip:

 

“Also I noticed the /var/lib/perfsonar/toolkit/gui-tasks.conf can sometimes get confused and needs to be reset... I didn't do a whole lot of digging into this part but "resetting" this file to the original testing parameters configured in the gui seems to clear any issues with disappearing tests.”

 

So I cleared gui-tasks.conf, submitted a single throughput test to luisteraar.nikhef.nl in the web interface, and then it worked!

 

The other problem, that a CLI test fails, still remains. Here’s the output with debug flags on. I tried to use --bind as we have a multi-homed system (two interfaces), but I get “no such option”.

 

[root@perfsonar ~]# pscheduler task --debug --debug-pscheduler throughput --dest luisteraar.nikhef.nl

2021-08-05T16:43:09 Debug started

2021-08-05T16:43:09 Assistance is from localhost

2021-08-05T16:43:09 Forcing default slip of PT5M

2021-08-05T16:43:09 Converting to spec via https://localhost/pscheduler/tests/throughput/spec

Submitting task...

2021-08-05T16:43:09 Fetching participant list

2021-08-05T16:43:09 Spec is: {"dest": "luisteraar.nikhef.nl", "schema": 1}

2021-08-05T16:43:09 Params are: {'spec': '{"dest": "luisteraar.nikhef.nl", "schema": 1}'}

2021-08-05T16:43:10 Got participants: {'participants': [None, 'luisteraar.nikhef.nl'], 'null-reason': 'No source specified'}

2021-08-05T16:43:10 Lead is None

2021-08-05T16:43:10 Null reason is No source specified

2021-08-05T16:43:10 Pinging https://localhost/pscheduler/

2021-08-05T16:43:10 localhost is up

2021-08-05T16:43:10 Posting task to https://localhost/pscheduler/tasks

2021-08-05T16:43:10 Data is {'schedule': {'slip': 'PT5M'}, 'test': {'spec': {'dest': 'luisteraar.nikhef.nl', 'schema': 1}, 'type': 'throughput'}, 'debug': True, 'schema': 4}

Task URL:

https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3

2021-08-05T16:43:16 Posted https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3

2021-08-05T16:43:16 Submission diagnostics:

2021-08-05T16:43:16   Hints:

2021-08-05T16:43:16     requester: ::1

2021-08-05T16:43:16     server: ::1

2021-08-05T16:43:16   Identified as everybody, local-interfaces

2021-08-05T16:43:16   Classified as default, friendlies

2021-08-05T16:43:16   Application: Hosts we trust to do everything

2021-08-05T16:43:16     Group 1: Limit 'always' passed

2021-08-05T16:43:16     Group 1: Want all, 1/1 passed, 0/1 failed: PASS

2021-08-05T16:43:16     Application PASSES

2021-08-05T16:43:16   Application: Defaults applied to non-friendly hosts

2021-08-05T16:43:16     Group 1: Limit 'innocuous-tests' failed: Passed but inverted

2021-08-05T16:43:16     Group 1: Limit 'throughput-default-time' passed

2021-08-05T16:43:16     Group 1: Limit 'idleex-default' failed: Test is not 'idleex'

2021-08-05T16:43:16     Group 1: Want any, 1/3 passed, 2/3 failed: PASS

2021-08-05T16:43:16     Application PASSES

2021-08-05T16:43:16   Proposal meets limits

2021-08-05T16:43:16   Priority set to default of 0

Running with tool 'iperf3'

Fetching first run...

2021-08-05T16:43:16 Fetching https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3/runs/first

2021-08-05T16:43:18 Handing off: pscheduler watch --first --debug https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3

2021-08-05T16:43:19 Debug started

2021-08-05T16:43:19 Fetching https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3

2021-08-05T16:43:19 Fetching next run from https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3/runs/first

 

Next scheduled run:

https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3/runs/d6ca1832-13af-402c-89d5-fa2095d8e1b2

Starts 2021-08-05T16:43:41+02 (~21 seconds)

Ends   2021-08-05T16:44:00+02 (~18 seconds)

Waiting for result...

 

2021-08-05T16:44:00 Calling ['pscheduler', 'result', '--quiet', '--format', 'text/plain', '--debug', 'https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3/runs/d6ca1832-13af-402c-89d5-fa2095d8e1b2']

Run failed.

 

Error:

  iperf3 returned an error: {

      "start":    {

            "connected":     [],

            "version": "iperf 3.9",

            "system_info":   "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"

      },

      "intervals":     [],

      "end":      {

      }

  }

  

  

  iperf3: error - unable to send control message: Bad file descriptor

  

  

 

Diagnostics:

  Participant 0:

  numactl -N 0 /usr/bin/iperf3 -p 5201 -c luisteraar.nikhef.nl -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmpnw4c6fmk/tmp73gikyj6/public-key --username lndyxWQlPKuj0NE3cUVa

  Participant 1:

  /usr/bin/iperf3 -s -1 --json -p 5201 --authorized-users-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/private-key

  

 

Limit system diagnostics for this run:

  Hints:

    requester: ::1

    server: ::1

  Identified as everybody, local-interfaces

  Classified as default, friendlies

  Application: Hosts we trust to do everything

    Group 1: Limit 'always' passed

    Group 1: Want all, 1/1 passed, 0/1 failed: PASS

    Application PASSES

  Application: Defaults applied to non-friendly hosts

    Group 1: Limit 'innocuous-tests' failed: Passed but inverted

    Group 1: Limit 'throughput-default-time' passed

    Group 1: Limit 'idleex-default' failed: Test is not 'idleex'

    Group 1: Want any, 1/3 passed, 2/3 failed: PASS

    Application PASSES

  Proposal meets limits

  Priority set to default of 0

 

Error from localhost:

  iperf3 returned an error: {

      "start":    {

            "connected":     [],

            "version": "iperf 3.9",

            "system_info":   "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"

      },

      "intervals":     [],

      "end":      {

      }

  }

  

  

  iperf3: error - unable to send control message: Bad file descriptor

  

  

 

Diagnostics from localhost:

  numactl -N 0 /usr/bin/iperf3 -p 5201 -c luisteraar.nikhef.nl -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmpnw4c6fmk/tmp73gikyj6/public-key --username lndyxWQlPKuj0NE3cUVa

 

Error from luisteraar.nikhef.nl:

  iperf3 returned an error: 

  

  Process took too long to run.

  

 

Diagnostics from luisteraar.nikhef.nl:

  /usr/bin/iperf3 -s -1 --json -p 5201 --authorized-users-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/private-key

2021-08-05T16:44:01 Fetching next run from https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3/runs/next

 

No further runs scheduled.

 

 

 

 

Van: "Garnizov, Ivan" <>
Datum: woensdag 4 augustus 2021 om 15:07
Aan: Onno Zweers <>, "" <>
Onderwerp: RE: [perfsonar-user] Added tests not listed in the Test Results

 

Hi Onno,

 

I am not sure, if my previous reply reached you…therefor adding it below again.

Please check the suggestion for the command you’d need to run when diagnosing an issue.

 

To this and with regards your last email, I can only say that in the first 2 outcomes, pScheduler didn’t succeed in coordinating a test between the 2 servers and in the third case the test was coordinated, but then failed during the iperf3 execution.

 

I might want to try run this same iperf3 test manually to better understand where the issue resides. Certainly it is a complex one, having in mind the first 2 results. It appears that besides other issues on iperf3 level, you might also be having these issues, because of your systems pS schedule utilisation.

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From: Garnizov, Ivan (RRZE)
Sent: Wednesday, August 4, 2021 10:20 AM
To: 'Onno Zweers' <>; perfsonar-user () <>
Subject: RE: Added tests not listed in the Test Results

 

Hi Onno,

 

“Does that mean our Perfsonar is too busy?”

No! Just from a single attempt it is not possible to make such conclusions.

It is not even clear, which side is busy. With throughout you have both ends, that should have a free time slot and match it for a this test to succeed.

 

You should make this same test with -- debug  and -- slip options:

 

pscheduler task --debug throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl --slip PT30M

 

for the utilisation of the pScheduler schedule you should look in http://docs.perfsonar.net/pscheduler_client_schedule.html#visualizing-the-schedule-with-pscheduler-plot-schedule

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

From: [mailto:] On Behalf Of Onno Zweers
Sent: Wednesday, August 4, 2021 10:19 AM
To:
Subject: Re: [perfsonar-user] Added tests not listed in the Test Results

 

Hi all,

 

I tried a few times from the command line, with three different results. Not sure what it all means.

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

Submitting task...

Task URL:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/3bc763d2-554d-4566-bc2a-ea0fe9f0fc61

Running with tool 'iperf3'

Fetching first run...

 

Next scheduled run:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/3bc763d2-554d-4566-bc2a-ea0fe9f0fc61/runs/cfead1c9-fc8a-481f-9aec-ef6d46db672b

 

Run scheduled at 2021-08-04T09:59:51+02 is a non-starter:

Gave up after too many scheduling conflicts.

 

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

Submitting task...

Task URL:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/a1a000f6-b185-4972-ad85-090c1e1df10d

Running with tool 'iperf3'

Fetching first run...

Unable to fetch runs: No such run.

 

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

Submitting task...

Task URL:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/6d704cf8-de1f-4ec8-b080-35e282680b02

Running with tool 'iperf3'

Fetching first run...

 

Next scheduled run:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/6d704cf8-de1f-4ec8-b080-35e282680b02/runs/bcf7f1ac-bcff-4ea5-9be3-0412fba7ad2c

Starts 2021-08-04T10:08:51+02 (~10 seconds)

Ends   2021-08-04T10:09:10+02 (~18 seconds)

Waiting for result...

 

Run failed.

 

Error:

  iperf3 returned an error: {

     "start":   {

          "connected":    [],

          "version": "iperf 3.9",

          "system_info":  "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"

     },

     "intervals":    [],

     "end":     {

     }

  }

  

  

  iperf3: error - unable to send control message: Bad file descriptor

  

  

 

Diagnostics:

  Participant 0:

  numactl -N 0 /usr/bin/iperf3 -p 5201 -B 2001:610:108:203a::32 -c 2a07:8504:120:e068::72 -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmp80cl8f6t/tmp0xaouzpa/public-key --username H0ZOxEmIfGkflgY1leYT

  Participant 1:

  /usr/bin/iperf3 -s -1 --json -p 5201 -B 2a07:8504:120:e068::72 --authorized-users-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/private-key

  

 

Limit system diagnostics for this run:

  Hints:

    requester: 2001:610:108:203a::32

    server: 2001:610:108:203a::32

  Identified as everybody, local-interfaces

  Classified as default, friendlies

  Application: Hosts we trust to do everything

    Group 1: Limit 'always' passed

    Group 1: Want all, 1/1 passed, 0/1 failed: PASS

    Application PASSES

  Application: Defaults applied to non-friendly hosts

    Group 1: Limit 'innocuous-tests' failed: Passed but inverted

    Group 1: Limit 'throughput-default-time' passed

    Group 1: Limit 'idleex-default' failed: Test is not 'idleex'

    Group 1: Want any, 1/3 passed, 2/3 failed: PASS

    Application PASSES

  Proposal meets limits

  Priority set to default of 0

 

Error from perfsonar-bandwidth.grid.surfsara.nl:

  iperf3 returned an error: {

     "start":   {

          "connected":    [],

          "version": "iperf 3.9",

          "system_info":  "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"

     },

     "intervals":    [],

     "end":     {

     }

  }

  

  

  iperf3: error - unable to send control message: Bad file descriptor

  

  

 

Diagnostics from perfsonar-bandwidth.grid.surfsara.nl:

  numactl -N 0 /usr/bin/iperf3 -p 5201 -B 2001:610:108:203a::32 -c 2a07:8504:120:e068::72 -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmp80cl8f6t/tmp0xaouzpa/public-key --username H0ZOxEmIfGkflgY1leYT

 

Error from luisteraar.nikhef.nl:

  iperf3 returned an error: 

  

  Process took too long to run.

  

 

Diagnostics from luisteraar.nikhef.nl:

  /usr/bin/iperf3 -s -1 --json -p 5201 -B 2a07:8504:120:e068::72 --authorized-users-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/private-key

 

No further runs scheduled.

 

 

Van: <> namens Onno Zweers <>
Beantwoorden - Aan: Onno Zweers <>
Datum: vrijdag 30 juli 2021 om 12:45
Aan: "" <>
Onderwerp: [perfsonar-user] Added tests not listed in the Test Results

 

Dear people,

 

My knowledge of Perfsonar is limited; apologies if I’m asking for something obvious. Our Perfsonar https://perfsonar-bandwidth.grid.surfsara.nl/toolkit/ is happily measuring lots of other WLCG Perfsonars. Now I tried to add tests for our neighbors, Nikhef: https://luisteraar.nikhef.nl/toolkit/. So I’ve added them like in this screen shot:

 

However, when I try to look up results in the Test Results, they are simply not listed (not even the next day). I looked also on the private dashboard. No sign of luisteraar.nikhef.nl. Why is it not listed? I can ping from our PS to theirs so I assume there should be at least one test that doesn’t fail, and even if tests fail I’d still expect to see them in the results.

 

Kind regards,

Onno