perfsonar-user - [perfsonar-user] throughput test timeouts
Subject: perfSONAR User Q&A and Other Discussion
List archive
- From: Michael Thomas <>
- To:
- Subject: [perfsonar-user] throughput test timeouts
- Date: Mon, 7 Mar 2022 15:39:30 -0600
- Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=caltech.edu; dmarc=pass action=none header.from=caltech.edu; dkim=pass header.d=caltech.edu; arc=none
- Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8gPCXk8xTdcQ9BAHe7XXYaDdFGdgcNWcdiDYEPij5nM=; b=XbsHyQ/1H38LTlwmboq3B8ZVN3EeTzzfT/hSPXbn3qNGBhyZdCRgeD1UnRUrojV+fP61HfZI0zmoQtggovMwuyZaZXWyry5fVMTH52Mv3Hj/2/qwTHAlQoBo2HZJOjDss3l21MQd8baz4D+5P0duxMPaaG0tPbEoiuiZqzonGw0NtyTVA4S8Ka9qyWh/yo4yupW50h2y2/xbEQ4/pb5Mjv6fDG9bteRHnM9mBCPp7d2XSwPOF8m7aUr/VjSDj0MhEG7yQN6iFvhTdTIivq1icL9QtLSSVwJmY0bUl+deAjRIhwV39DL/MLut14vZ/RuoYsuZvGJdwmg8yxvn3RHKVg==
- Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z5qxSrnVp5x7R+7vV7CkGRvs9ZYhisHoXy/G5O47HN+rWvPnubYosPRhXAFsk5veihdwOztz4p+5C6LVgCplrNvayaGBkR2Ig1eCWZZTFnMZlcmRJPW+KUC4z4GsQfSWRGl1vN5SkWYxv8N3QhnAwh7Qi4k61a7ocXLJXD6+xvnmS8RfefzMXusiaBGrJczO31R5jv9ijbSwfFGt4px7yH5tv3DqMwD4sZRRpsIwsZnXhJpIX6E6z+/fz9+4FpnfvaNW6l3gghbOISqhEgvGtE70lqqRJgToLWpRthLNvpZL4tHYj+Rsx0oIW4td/IuQT+eXqkt7L+0eM2grH5yykA==
I recently set up a perfsonar server and configured it to run throughput tests with two other sites in our group. The iperf3 tests from my site to the remote locations works fine, but all incoming tests fail, with the following results in my pscheduler logs:
Mar 7 12:19:48 perfsonar.ligo-la.caltech.edu journal: runner INFO 12832: Running https://perfsonar-gc.ligo.caltech.edu/pscheduler/tasks/274d650f-166f-40bd-babe-c5189e16fda9/runs/a68ec259-b011-48a1-93a6-3be2879293cf
Mar 7 12:19:48 perfsonar.ligo-la.caltech.edu journal: runner INFO 12832: With iperf3: throughput --source perfsonar-gc.ligo.caltech.edu --source-node perfsonar-gc.ligo.caltech.edu --dest 208.69.128.92 --dest-node 208.69.128.92 --duration PT20S --ip-version 4
Mar 7 12:20:21 perfsonar.ligo-la.caltech.edu journal: runner INFO 12832: Run failed 2: Process took too long to run.
Testing from the CLI shows an authorization failure:
Error from perfsonar.ligo-wa.caltech.edu:
iperf3 output has neither 'sum_sent' nor 'sum' field, and one of them is required
Diagnostics from perfsonar.ligo-wa.caltech.edu:
/usr/bin/iperf3 -p 5201 -B 198.129.208.245 -c 208.69.128.92 -t 10 --json --rsa -public-key-path /var/run/pscheduler-server/runner/tmp/tmp_j91baqu/tmpumiughjb/public-key --username <username redacted>
{
"start": {
"connected": [],
"version": "iperf 3.11",
"system_info": "Linux perfsonar 4.9.0-17-amd64 #1 SMP Debian 4.
9.290-1 (2021-12-12) x86_64",
"timestamp": {
"time": "Mon, 07 Mar 2022 21:01:06 GMT",
"timesecs": 1646686866
},
"connecting_to": {
"host": "208.69.128.92",
"port": 5201
},
"cookie": "<cookie redacted>",
"tcp_mss_default": 8948,
"target_bitrate": 0
},
"intervals": [],
"end": {
},
"error": "test authorization failed"
}
How can I further debug this authorization failure?
I will also note that this looks suspiciously like the same issue described in a previous post, but killing the rogue iperf3 process does not allow subsequent tests to work.
https://lists.internet2.edu/sympa/arc/perfsonar-user/2021-08/msg00011.html
--Mike
- [perfsonar-user] throughput test timeouts, Michael Thomas, 03/07/2022
- RE: [perfsonar-user] throughput test timeouts, Garnizov, Ivan, 03/08/2022
- Re: [perfsonar-user] throughput test timeouts, Michael Thomas, 03/08/2022
- RE: [perfsonar-user] throughput test timeouts, Garnizov, Ivan, 03/11/2022
- Re: [perfsonar-user] throughput test timeouts, Mark Feit, 03/11/2022
- Re: [perfsonar-user] throughput test timeouts, Michael Thomas, 03/08/2022
- Re: [perfsonar-user] throughput test timeouts, Mark Feit, 03/08/2022
- Re: [perfsonar-user] throughput test timeouts, Michael Thomas, 03/08/2022
- RE: [perfsonar-user] throughput test timeouts, Garnizov, Ivan, 03/08/2022
Archive powered by MHonArc 2.6.24.