perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan" <>]

Hi Onno,

 

I am not sure, if my previous reply reached you…therefor adding it below again.

Please check the suggestion for the command you’d need to run when diagnosing an issue.

 

To this and with regards your last email, I can only say that in the first 2 outcomes, pScheduler didn’t succeed in coordinating a test between the 2 servers and in the third case the test was coordinated, but then failed during the iperf3 execution.

 

I might want to try run this same iperf3 test manually to better understand where the issue resides. Certainly it is a complex one, having in mind the first 2 results. It appears that besides other issues on iperf3 level, you might also be having these issues, because of your systems pS schedule utilisation.

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From: Garnizov, Ivan (RRZE)
Sent: Wednesday, August 4, 2021 10:20 AM
To: 'Onno Zweers' <>; perfsonar-user () <>
Subject: RE: Added tests not listed in the Test Results

 

Hi Onno,

 

“Does that mean our Perfsonar is too busy?”

No! Just from a single attempt it is not possible to make such conclusions.

It is not even clear, which side is busy. With throughout you have both ends, that should have a free time slot and match it for a this test to succeed.

 

You should make this same test with -- debug  and -- slip options:

 

pscheduler task --debug throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl --slip PT30M

 

for the utilisation of the pScheduler schedule you should look in http://docs.perfsonar.net/pscheduler_client_schedule.html#visualizing-the-schedule-with-pscheduler-plot-schedule

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

From: [mailto:] On Behalf Of Onno Zweers
Sent: Wednesday, August 4, 2021 10:19 AM
To:
Subject: Re: [perfsonar-user] Added tests not listed in the Test Results

 

Hi all,

 

I tried a few times from the command line, with three different results. Not sure what it all means.

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

Submitting task...

Task URL:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/3bc763d2-554d-4566-bc2a-ea0fe9f0fc61

Running with tool 'iperf3'

Fetching first run...

 

Next scheduled run:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/3bc763d2-554d-4566-bc2a-ea0fe9f0fc61/runs/cfead1c9-fc8a-481f-9aec-ef6d46db672b

 

Run scheduled at 2021-08-04T09:59:51+02 is a non-starter:

Gave up after too many scheduling conflicts.

 

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

Submitting task...

Task URL:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/a1a000f6-b185-4972-ad85-090c1e1df10d

Running with tool 'iperf3'

Fetching first run...

Unable to fetch runs: No such run.

 

 

[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl

Submitting task...

Task URL:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/6d704cf8-de1f-4ec8-b080-35e282680b02

Running with tool 'iperf3'

Fetching first run...

 

Next scheduled run:

https://perfsonar-bandwidth.grid.surfsara.nl/pscheduler/tasks/6d704cf8-de1f-4ec8-b080-35e282680b02/runs/bcf7f1ac-bcff-4ea5-9be3-0412fba7ad2c

Starts 2021-08-04T10:08:51+02 (~10 seconds)

Ends   2021-08-04T10:09:10+02 (~18 seconds)

Waiting for result...

 

Run failed.

 

Error:

  iperf3 returned an error: {

     "start":   {

          "connected":    [],

          "version": "iperf 3.9",

          "system_info":  "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"

     },

     "intervals":    [],

     "end":     {

     }

  }

  

  

  iperf3: error - unable to send control message: Bad file descriptor

  

  

 

Diagnostics:

  Participant 0:

  numactl -N 0 /usr/bin/iperf3 -p 5201 -B 2001:610:108:203a::32 -c 2a07:8504:120:e068::72 -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmp80cl8f6t/tmp0xaouzpa/public-key --username H0ZOxEmIfGkflgY1leYT

  Participant 1:

  /usr/bin/iperf3 -s -1 --json -p 5201 -B 2a07:8504:120:e068::72 --authorized-users-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/private-key

  

 

Limit system diagnostics for this run:

  Hints:

    requester: 2001:610:108:203a::32

    server: 2001:610:108:203a::32

  Identified as everybody, local-interfaces

  Classified as default, friendlies

  Application: Hosts we trust to do everything

    Group 1: Limit 'always' passed

    Group 1: Want all, 1/1 passed, 0/1 failed: PASS

    Application PASSES

  Application: Defaults applied to non-friendly hosts

    Group 1: Limit 'innocuous-tests' failed: Passed but inverted

    Group 1: Limit 'throughput-default-time' passed

    Group 1: Limit 'idleex-default' failed: Test is not 'idleex'

    Group 1: Want any, 1/3 passed, 2/3 failed: PASS

    Application PASSES

  Proposal meets limits

  Priority set to default of 0

 

Error from perfsonar-bandwidth.grid.surfsara.nl:

  iperf3 returned an error: {

     "start":   {

          "connected":    [],

          "version": "iperf 3.9",

          "system_info":  "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"

     },

     "intervals":    [],

     "end":     {

     }

  }

  

  

  iperf3: error - unable to send control message: Bad file descriptor

  

  

 

Diagnostics from perfsonar-bandwidth.grid.surfsara.nl:

  numactl -N 0 /usr/bin/iperf3 -p 5201 -B 2001:610:108:203a::32 -c 2a07:8504:120:e068::72 -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmp80cl8f6t/tmp0xaouzpa/public-key --username H0ZOxEmIfGkflgY1leYT

 

Error from luisteraar.nikhef.nl:

  iperf3 returned an error: 

  

  Process took too long to run.

  

 

Diagnostics from luisteraar.nikhef.nl:

  /usr/bin/iperf3 -s -1 --json -p 5201 -B 2a07:8504:120:e068::72 --authorized-users-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/private-key

 

No further runs scheduled.

 

 

Van: <> namens Onno Zweers <>
Beantwoorden - Aan: Onno Zweers <>
Datum: vrijdag 30 juli 2021 om 12:45
Aan: "" <>
Onderwerp: [perfsonar-user] Added tests not listed in the Test Results

 

Dear people,

 

My knowledge of Perfsonar is limited; apologies if I’m asking for something obvious. Our Perfsonar https://perfsonar-bandwidth.grid.surfsara.nl/toolkit/ is happily measuring lots of other WLCG Perfsonars. Now I tried to add tests for our neighbors, Nikhef: https://luisteraar.nikhef.nl/toolkit/. So I’ve added them like in this screen shot:

 

However, when I try to look up results in the Test Results, they are simply not listed (not even the next day). I looked also on the private dashboard. No sign of luisteraar.nikhef.nl. Why is it not listed? I can ping from our PS to theirs so I assume there should be at least one test that doesn’t fail, and even if tests fail I’d still expect to see them in the results.

 

Kind regards,

Onno