perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan" <>]

Hello Michael,

 

This really sounds strange. I would suggest you to collect some diagnostic info from the CLI output and report back the results. We need to find a way reproduce your issue.

Please recollect all the output of these commands, ran one after the other

 

ps aux

sudo kill <<iperf3>>

ps aux

pscheduler task throughput .....

sleep 10

ps aux

 

Please note the kill and pscheduler commands should be adjusted accordingly to kill an iperf3 process and then to produce a valid throughput test, regardless if it succeeds or not, we need to see the iperf3 tool started.

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

 

 

-----Original Message-----
From: Michael Thomas []
Sent: Tuesday, March 8, 2022 6:59 PM
To: Garnizov, Ivan (RRZE) <>;
Subject: Re: [perfsonar-user] throughput test timeouts

 

Hi Ivan,

 

On 3/8/22 06:41, Garnizov, Ivan wrote:

> Hello Michael,

>

> IMO, also, your case appears similar to the issue discussed in August

> (the link you refer to) Please tell about the versions of pS on both sides, or at least make sure both run 4.4.2 and above.

 

All hosts are running perfsonar 4.4.3

 

> Your comment about the killing the rogue iperf3 process does not hold water. Since iperf3 tool is started and stopped with each pscheduler throughput run.

> If on your side iperf3 operation is different, then we need to investigate this.

 

This is in fact the problematic behavior.  The iperf3 process does not go away after each throughput test.  It does appear that the stray

iperf3 process gets killed when the next throughput test starts, but the test will launch a new iperf3 process that stays around after the test is complete.

 

--Mike

 

> Of course killing iperf3 during the test also won't help.

>

> Regards,

> Ivan Garnizov

>

> GEANT WP6T3: pS development team

> GEANT WP7T1: pS deployments GN Operations GEANT WP9T2: Software

> governance in GEANT

>

>

>

>

> -----Original Message-----

> From:

> [] On Behalf Of Michael

> Thomas

> Sent: Monday, March 7, 2022 10:40 PM

> To:

> Subject: [perfsonar-user] throughput test timeouts

>

> I recently set up a perfsonar server and configured it to run throughput tests with two other sites in our group.  The iperf3 tests from my site to the remote locations works fine, but all incoming tests fail, with the following results in my pscheduler logs:

>

> Mar  7 12:19:48 perfsonar.ligo-la.caltech.edu journal: runner INFO

> 12832: Running

> https://perfsonar-gc.ligo.caltech.edu/pscheduler/tasks/274d650f-166f-4

> 0bd-babe-c5189e16fda9/runs/a68ec259-b011-48a1-93a6-3be2879293cf

> Mar  7 12:19:48 perfsonar.ligo-la.caltech.edu journal: runner INFO

> 12832: With iperf3: throughput --source perfsonar-gc.ligo.caltech.edu

> --source-node perfsonar-gc.ligo.caltech.edu --dest 208.69.128.92

> --dest-node 208.69.128.92 --duration PT20S --ip-version 4 Mar  7

> 12:20:21 perfsonar.ligo-la.caltech.edu journal: runner INFO

> 12832: Run failed 2: Process took too long to run.

>

> Testing from the CLI shows an authorization failure:

>

> Error from perfsonar.ligo-wa.caltech.edu:

>     iperf3 output has neither 'sum_sent' nor 'sum' field, and one of

> them is required

>

> Diagnostics from perfsonar.ligo-wa.caltech.edu:

>     /usr/bin/iperf3 -p 5201 -B 198.129.208.245 -c 208.69.128.92 -t 10

> --json --rsa -public-key-path

> /var/run/pscheduler-server/runner/tmp/tmp_j91baqu/tmpumiughjb/public-k

> ey

> --username <username redacted>

>

>     {

>           "start":        {

>                   "connected":    [],

>                   "version":      "iperf 3.11",

>                   "system_info":  "Linux perfsonar 4.9.0-17-amd64 #1 SMP Debian 4.

> 9.290-1 (2021-12-12) x86_64",

>                   "timestamp":    {

>                           "time": "Mon, 07 Mar 2022 21:01:06 GMT",

>                           "timesecs":     1646686866

>                   },

>                   "connecting_to":        {

>                           "host": "208.69.128.92",

>                           "port": 5201

>                   },

>                   "cookie":       "<cookie redacted>",

>                   "tcp_mss_default":      8948,

>                   "target_bitrate":       0

>           },

>           "intervals":    [],

>           "end":  {

>           },

>           "error":        "test authorization failed"

>     }

>

> How can I further debug this authorization failure?

>

> I will also note that this looks suspiciously like the same issue described in a previous post, but killing the rogue iperf3 process does not allow subsequent tests to work.

>

> https://lists.internet2.edu/sympa/arc/perfsonar-user/2021-08/msg00011.

> html

>

> --Mike