perfSONAR User Q&A and Other Discussion

[Michael Thomas <>]

Hi Ivan,

On 3/8/22 06:41, Garnizov, Ivan wrote:
> Hello Michael,
>
> IMO, also, your case appears similar to the issue discussed in August (the 
> link you refer to)
> Please tell about the versions of pS on both sides, or at least make sure 
> both run 4.4.2 and above.

All hosts are running perfsonar 4.4.3

> Your comment about the killing the rogue iperf3 process does not hold water. 
> Since iperf3 tool is started and stopped with each pscheduler throughput run.
> If on your side iperf3 operation is different, then we need to investigate 
> this.

This is in fact the problematic behavior.  The iperf3 process does not 
go away after each throughput test.  It does appear that the stray 
iperf3 process gets killed when the next throughput test starts, but the 
test will launch a new iperf3 process that stays around after the test 
is complete.

--Mike

> Of course killing iperf3 during the test also won't help.
>
> Regards,
> Ivan Garnizov
>
> GEANT WP6T3: pS development team
> GEANT WP7T1: pS deployments GN Operations
> GEANT WP9T2: Software governance in GEANT
>
>
>
>
> -----Original Message-----
> From:  
> [] On Behalf Of Michael Thomas
> Sent: Monday, March 7, 2022 10:40 PM
> To: 
> Subject: [perfsonar-user] throughput test timeouts
>
> I recently set up a perfsonar server and configured it to run throughput 
> tests with two other sites in our group.  The iperf3 tests from my site to 
> the remote locations works fine, but all incoming tests fail, with the 
> following results in my pscheduler logs:
>
> Mar  7 12:19:48 perfsonar.ligo-la.caltech.edu journal: runner INFO
> 12832: Running
> https://perfsonar-gc.ligo.caltech.edu/pscheduler/tasks/274d650f-166f-40bd-babe-c5189e16fda9/runs/a68ec259-b011-48a1-93a6-3be2879293cf
> Mar  7 12:19:48 perfsonar.ligo-la.caltech.edu journal: runner INFO
> 12832: With iperf3: throughput --source perfsonar-gc.ligo.caltech.edu 
> --source-node perfsonar-gc.ligo.caltech.edu --dest 208.69.128.92 --dest-node 
> 208.69.128.92 --duration PT20S --ip-version 4 Mar  7 12:20:21 
> perfsonar.ligo-la.caltech.edu journal: runner INFO
> 12832: Run failed 2: Process took too long to run.
>
> Testing from the CLI shows an authorization failure:
>
> Error from perfsonar.ligo-wa.caltech.edu:
>     iperf3 output has neither 'sum_sent' nor 'sum' field, and one of them is 
> required
>
> Diagnostics from perfsonar.ligo-wa.caltech.edu:
>     /usr/bin/iperf3 -p 5201 -B 198.129.208.245 -c 208.69.128.92 -t 10 --json 
> --rsa -public-key-path 
> /var/run/pscheduler-server/runner/tmp/tmp_j91baqu/tmpumiughjb/public-key
> --username <username redacted>
>
>     {
>           "start":        {
>                   "connected":    [],
>                   "version":      "iperf 3.11",
>                   "system_info":  "Linux perfsonar 4.9.0-17-amd64 #1 SMP 
> Debian 4.
> 9.290-1 (2021-12-12) x86_64",
>                   "timestamp":    {
>                           "time": "Mon, 07 Mar 2022 21:01:06 GMT",
>                           "timesecs":     1646686866
>                   },
>                   "connecting_to":        {
>                           "host": "208.69.128.92",
>                           "port": 5201
>                   },
>                   "cookie":       "<cookie redacted>",
>                   "tcp_mss_default":      8948,
>                   "target_bitrate":       0
>           },
>           "intervals":    [],
>           "end":  {
>           },
>           "error":        "test authorization failed"
>     }
>
> How can I further debug this authorization failure?
>
> I will also note that this looks suspiciously like the same issue described 
> in a previous post, but killing the rogue iperf3 process does not allow 
> subsequent tests to work.
>
> https://lists.internet2.edu/sympa/arc/perfsonar-user/2021-08/msg00011.html
>
> --Mike