Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Added tests not listed in the Test Results

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Added tests not listed in the Test Results


Chronological Thread 
    < Chronological >      < Thread >
  • From: Antoine Delvaux <>
  • To: Onno Zweers <>
  • Cc: Ivan Garnizov <>, "" <>
  • Subject: Re: [perfsonar-user] Added tests not listed in the Test Results
  • Date: Mon, 9 Aug 2021 19:32:47 +0100
Hi Onno,

Trying to run a throughput task with iperf3 to luisteraar.nikhef.nl gives me the same error message as you received.  It seems it is on luisteraar.nikhef.nl that there is an issue.  Maybe there is more info in the logs of this machine?

What is puzzling though is that your web UI is showing results, so some tests are running.  Maybe the ones configured from the other side?

Regards,

Antoine.

Le 5 août 2021 à 16:05, Onno Zweers <> a écrit :

Hi all,
 
One of my problems has been fixed. Mary Hester @ Nikhef had a tip:
 
“Also I noticed the /var/lib/perfsonar/toolkit/gui-tasks.conf can sometimes get confused and needs to be reset... I didn't do a whole lot of digging into this part but "resetting" this file to the original testing parameters configured in the gui seems to clear any issues with disappearing tests.”
 
So I cleared gui-tasks.conf, submitted a single throughput test to luisteraar.nikhef.nl in the web interface, and then it worked!
 
The other problem, that a CLI test fails, still remains. Here’s the output with debug flags on. I tried to use --bind as we have a multi-homed system (two interfaces), but I get “no such option”.
 
[root@perfsonar ~]# pscheduler task --debug --debug-pscheduler throughput --dest luisteraar.nikhef.nl
2021-08-05T16:43:09 Debug started
2021-08-05T16:43:09 Assistance is from localhost
2021-08-05T16:43:09 Forcing default slip of PT5M
2021-08-05T16:43:09 Converting to spec via https://localhost/pscheduler/tests/throughput/spec
Submitting task...
2021-08-05T16:43:09 Fetching participant list
2021-08-05T16:43:09 Spec is: {"dest": "luisteraar.nikhef.nl", "schema": 1}
2021-08-05T16:43:09 Params are: {'spec': '{"dest": "luisteraar.nikhef.nl", "schema": 1}'}
2021-08-05T16:43:10 Got participants: {'participants': [None, 'luisteraar.nikhef.nl'], 'null-reason': 'No source specified'}
2021-08-05T16:43:10 Lead is None
2021-08-05T16:43:10 Null reason is No source specified
2021-08-05T16:43:10 Pinging https://localhost/pscheduler/
2021-08-05T16:43:10 localhost is up
2021-08-05T16:43:10 Posting task to https://localhost/pscheduler/tasks
2021-08-05T16:43:10 Data is {'schedule': {'slip': 'PT5M'}, 'test': {'spec': {'dest': 'luisteraar.nikhef.nl', 'schema': 1}, 'type': 'throughput'}, 'debug': True, 'schema': 4}
Task URL:
2021-08-05T16:43:16 Submission diagnostics:
2021-08-05T16:43:16   Hints:
2021-08-05T16:43:16     requester: ::1
2021-08-05T16:43:16     server: ::1
2021-08-05T16:43:16   Identified as everybody, local-interfaces
2021-08-05T16:43:16   Classified as default, friendlies
2021-08-05T16:43:16   Application: Hosts we trust to do everything
2021-08-05T16:43:16     Group 1: Limit 'always' passed
2021-08-05T16:43:16     Group 1: Want all, 1/1 passed, 0/1 failed: PASS
2021-08-05T16:43:16     Application PASSES
2021-08-05T16:43:16   Application: Defaults applied to non-friendly hosts
2021-08-05T16:43:16     Group 1: Limit 'innocuous-tests' failed: Passed but inverted
2021-08-05T16:43:16     Group 1: Limit 'throughput-default-time' passed
2021-08-05T16:43:16     Group 1: Limit 'idleex-default' failed: Test is not 'idleex'
2021-08-05T16:43:16     Group 1: Want any, 1/3 passed, 2/3 failed: PASS
2021-08-05T16:43:16     Application PASSES
2021-08-05T16:43:16   Proposal meets limits
2021-08-05T16:43:16   Priority set to default of 0
Running with tool 'iperf3'
Fetching first run...
2021-08-05T16:43:18 Handing off: pscheduler watch --first --debug https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3
2021-08-05T16:43:19 Debug started
 
Next scheduled run:
Starts 2021-08-05T16:43:41+02 (~21 seconds)
Ends   2021-08-05T16:44:00+02 (~18 seconds)
Waiting for result...
 
2021-08-05T16:44:00 Calling ['pscheduler', 'result', '--quiet', '--format', 'text/plain', '--debug', 'https://localhost/pscheduler/tasks/dad39b19-0f86-4695-b9b6-abe4bac861f3/runs/d6ca1832-13af-402c-89d5-fa2095d8e1b2']
Run failed.
 
Error:
  iperf3 returned an error: {
      "start":    {
            "connected":     [],
            "version": "iperf 3.9",
            "system_info":   "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"
      },
      "intervals":     [],
      "end":      {
      }
  }
  
  
  iperf3: error - unable to send control message: Bad file descriptor
  
  
 
Diagnostics:
  Participant 0:
  numactl -N 0 /usr/bin/iperf3 -p 5201 -c luisteraar.nikhef.nl -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmpnw4c6fmk/tmp73gikyj6/public-key --username lndyxWQlPKuj0NE3cUVa
  Participant 1:
  /usr/bin/iperf3 -s -1 --json -p 5201 --authorized-users-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/private-key
  
 
Limit system diagnostics for this run:
  Hints:
    requester: ::1
    server: ::1
  Identified as everybody, local-interfaces
  Classified as default, friendlies
  Application: Hosts we trust to do everything
    Group 1: Limit 'always' passed
    Group 1: Want all, 1/1 passed, 0/1 failed: PASS
    Application PASSES
  Application: Defaults applied to non-friendly hosts
    Group 1: Limit 'innocuous-tests' failed: Passed but inverted
    Group 1: Limit 'throughput-default-time' passed
    Group 1: Limit 'idleex-default' failed: Test is not 'idleex'
    Group 1: Want any, 1/3 passed, 2/3 failed: PASS
    Application PASSES
  Proposal meets limits
  Priority set to default of 0
 
Error from localhost:
  iperf3 returned an error: {
      "start":    {
            "connected":     [],
            "version": "iperf 3.9",
            "system_info":   "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"
      },
      "intervals":     [],
      "end":      {
      }
  }
  
  
  iperf3: error - unable to send control message: Bad file descriptor
  
  
 
Diagnostics from localhost:
  numactl -N 0 /usr/bin/iperf3 -p 5201 -c luisteraar.nikhef.nl -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmpnw4c6fmk/tmp73gikyj6/public-key --username lndyxWQlPKuj0NE3cUVa
 
  iperf3 returned an error: 
  
  Process took too long to run.
  
 
Diagnostics from luisteraar.nikhef.nl:
  /usr/bin/iperf3 -s -1 --json -p 5201 --authorized-users-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmp5ilg2kq_/tmp3mb7p6ij/private-key
 
No further runs scheduled.
 
 
 
 
Van: "Garnizov, Ivan" <>
Datum: woensdag 4 augustus 2021 om 15:07
Aan: Onno Zweers <>, "" <>
Onderwerp: RE: [perfsonar-user] Added tests not listed in the Test Results
 
Hi Onno,
 
I am not sure, if my previous reply reached you…therefor adding it below again.
Please check the suggestion for the command you’d need to run when diagnosing an issue.
 
To this and with regards your last email, I can only say that in the first 2 outcomes, pScheduler didn’t succeed in coordinating a test between the 2 servers and in the third case the test was coordinated, but then failed during the iperf3 execution.
 
I might want to try run this same iperf3 test manually to better understand where the issue resides. Certainly it is a complex one, having in mind the first 2 results. It appears that besides other issues on iperf3 level, you might also be having these issues, because of your systems pS schedule utilisation.
 
 
Regards,
Ivan Garnizov
 
GEANT WP6T3: pS development team
GEANT WP7T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
 
 
 
From: Garnizov, Ivan (RRZE) 
Sent: Wednesday, August 4, 2021 10:20 AM
To: 'Onno Zweers' <>; perfsonar-user () <>
Subject: RE: Added tests not listed in the Test Results
 
Hi Onno,
 
“Does that mean our Perfsonar is too busy?”
No! Just from a single attempt it is not possible to make such conclusions.
It is not even clear, which side is busy. With throughout you have both ends, that should have a free time slot and match it for a this test to succeed.
 
You should make this same test with -- debug  and -- slip options:
 
pscheduler task --debug throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl --slip PT30M
 
 
 
Regards,
Ivan Garnizov
 
GEANT WP6T3: pS development team
GEANT WP7T1: pS deployments GN Operations
GEANT WP9T2: Software governance in GEANT
 
 
From:  [] On Behalf Of Onno Zweers
Sent: Wednesday, August 4, 2021 10:19 AM
To: 
Subject: Re: [perfsonar-user] Added tests not listed in the Test Results
 
Hi all,
 
I tried a few times from the command line, with three different results. Not sure what it all means.
 
[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl
Submitting task...
Task URL:
Running with tool 'iperf3'
Fetching first run...
 
Next scheduled run:
 
Run scheduled at 2021-08-04T09:59:51+02 is a non-starter:
Gave up after too many scheduling conflicts.
 
 
[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl
Submitting task...
Task URL:
Running with tool 'iperf3'
Fetching first run...
Unable to fetch runs: No such run.
 
 
[root@perfsonar ~]# pscheduler task throughput --source perfsonar-bandwidth.grid.surfsara.nl --dest luisteraar.nikhef.nl
Submitting task...
Task URL:
Running with tool 'iperf3'
Fetching first run...
 
Next scheduled run:
Starts 2021-08-04T10:08:51+02 (~10 seconds)
Ends   2021-08-04T10:09:10+02 (~18 seconds)
Waiting for result...
 
Run failed.
 
Error:
  iperf3 returned an error: {
     "start":   {
          "connected":    [],
          "version": "iperf 3.9",
          "system_info":  "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"
     },
     "intervals":    [],
     "end":     {
     }
  }
  
  
  iperf3: error - unable to send control message: Bad file descriptor
  
  
 
Diagnostics:
  Participant 0:
  numactl -N 0 /usr/bin/iperf3 -p 5201 -B 2001:610:108:203a::32 -c 2a07:8504:120:e068::72 -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmp80cl8f6t/tmp0xaouzpa/public-key --username H0ZOxEmIfGkflgY1leYT
  Participant 1:
  /usr/bin/iperf3 -s -1 --json -p 5201 -B 2a07:8504:120:e068::72 --authorized-users-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/private-key
  
 
Limit system diagnostics for this run:
  Hints:
    requester: 2001:610:108:203a::32
    server: 2001:610:108:203a::32
  Identified as everybody, local-interfaces
  Classified as default, friendlies
  Application: Hosts we trust to do everything
    Group 1: Limit 'always' passed
    Group 1: Want all, 1/1 passed, 0/1 failed: PASS
    Application PASSES
  Application: Defaults applied to non-friendly hosts
    Group 1: Limit 'innocuous-tests' failed: Passed but inverted
    Group 1: Limit 'throughput-default-time' passed
    Group 1: Limit 'idleex-default' failed: Test is not 'idleex'
    Group 1: Want any, 1/3 passed, 2/3 failed: PASS
    Application PASSES
  Proposal meets limits
  Priority set to default of 0
 
  iperf3 returned an error: {
     "start":   {
          "connected":    [],
          "version": "iperf 3.9",
          "system_info":  "Linux perfsonar.grid.surfsara.nl 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64"
     },
     "intervals":    [],
     "end":     {
     }
  }
  
  
  iperf3: error - unable to send control message: Bad file descriptor
  
  
 
  numactl -N 0 /usr/bin/iperf3 -p 5201 -B 2001:610:108:203a::32 -c 2a07:8504:120:e068::72 -t 10 --json --rsa-public-key-path /run/pscheduler-server/runner/tmp/tmp80cl8f6t/tmp0xaouzpa/public-key --username H0ZOxEmIfGkflgY1leYT
 
  iperf3 returned an error: 
  
  Process took too long to run.
  
 
Diagnostics from luisteraar.nikhef.nl:
  /usr/bin/iperf3 -s -1 --json -p 5201 -B 2a07:8504:120:e068::72 --authorized-users-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/credentials --rsa-private-key-path /run/pscheduler-server/runner/tmp/tmpv1h9njl6/tmpcsmojn_f/private-key
 
No further runs scheduled.
 
 
Van: <> namens Onno Zweers <>
Beantwoorden - Aan: Onno Zweers <>
Datum: vrijdag 30 juli 2021 om 12:45
Aan: "" <>
Onderwerp: [perfsonar-user] Added tests not listed in the Test Results
 
Dear people,
 
My knowledge of Perfsonar is limited; apologies if I’m asking for something obvious. Our Perfsonarhttps://perfsonar-bandwidth.grid.surfsara.nl/toolkit/ is happily measuring lots of other WLCG Perfsonars. Now I tried to add tests for our neighbors, Nikhef: https://luisteraar.nikhef.nl/toolkit/. So I’ve added them like in this screen shot:
 
<image001.png>
However, when I try to look up results in the Test Results, they are simply not listed (not even the next day). I looked also on the private dashboard. No sign of luisteraar.nikhef.nl. Why is it not listed? I can ping from our PS to theirs so I assume there should be at least one test that doesn’t fail, and even if tests fail I’d still expect to see them in the results.
 
Kind regards,
Onno
--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user


Archive powered by MHonArc 2.6.24.

Top of Page