perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan" <>]

Hello George,

 

I doubt there is a direct relation between the pSconfig event and the frequency of throughput failures.

I would suggest checking /var/log/perfsonar/psconfig-pscheduler-agent-tasks.log to look for missing records ;)

You know for tests between the hosts in question.

 

Please also check the pScheduler schedule for Non-Starters of runs on this test specification.

 

Please tell, if these servers are all of pS 4.4 release or is there a mixture of 4.4 and 4.3.x?

 

Please also note that the test you shared doesn’t match the direction of the test failure from above.

Failure is: HostB -> HostA

Success is: HostA -> HostB

 

The error you share tells me only that pSconfig wasn’t able to remove a stalled record. This by itself might in fact lead indeed to problems. I’ll leave to other for a comment.

 

 

Regards,

Ivan Garnizov

 

GEANT WP6T3: pS development team

GEANT WP7T1: pS deployments GN Operations

GEANT WP9T2: Software governance in GEANT

 

 

 

From: [mailto:] On Behalf Of "Uhl, George D. (GSFC-423.0)[Arctic Slope Technical Services, Inc.]"
Sent: Tuesday, August 3, 2021 3:22 PM
To:
Cc: Jackson, Wayne <>; Germain, Andrew M. (GSFC-423.0)[Arctic Slope Technical Services, Inc.] <>; Butler, Todd F <>
Subject: [perfsonar-user] Cannot query non-lead server for run result

 

Hello,

 

We’ve been experiencing multiple test failures since upgrading from 4.3.4 to 4.4.0.  These failures have impacted the regular testing within our mesh.

 

Our mesh throughput tests are scheduled to run in a 4 hour, 2 hour window or a 1 hour window. Slip time for tests are allotted the same time windows and slip time is randomized. We are getting the reoccurring error we’re receiving in tests scheduled through the mesh. 

 

Cannot query non-lead server for run result.

 

We think the following error in psconfig-pscheduler-agent-transactions.log might be a clue but we don’t understand its intent.

 

2021/08/03 12:27:45 ERROR guid=8B4895F0-F455-11EB-BBAB-A4BBA32715F6 action="delete" msg=Problem deleting test throughput/iperf3([Test-Host-B IP]->[Test-Host-A IP]), continuing with rest of config: FORBIDDEN: Forbidden.

 

However, we’re able to run adhoc throughput tests with these same hosts on the CLI with no problem. See below.

 

Thanks,

George Uhl

NASA GSFC

 

# pscheduler task throughput --source <managed-host-A IP> --source-node <managed-host-A IP> --dest <managed-host-B IP> --dest-node <managed-host-B IP> --duration PT30S --ip-version 4

Submitting task...

Task URL:

https:// <managed-host-A IP>/pscheduler/tasks/7bb99a7f-e682-48c3-8fad-83ee5f1c776e

Running with tool 'iperf3'

Fetching first run...

 

Next scheduled run:

https:// <managed-host-A IP>/pscheduler/tasks/7bb99a7f-e682-48c3-8fad-83ee5f1c776e/runs/c0853f4a-ace3-4e71-8005-def380e2f626

Starts 2021-08-02T10:38:26-04 (~5 seconds)

Ends   2021-08-02T10:39:05-04 (~38 seconds)

Waiting for result...

 

* Stream ID 5

Interval       Throughput     Retransmits    Current Window

0.0 - 1.0      22.75 Mbps     0              441.64 KBytes

1.0 - 2.0      125.81 Mbps    0              3.75 MBytes

2.0 - 3.0      157.28 Mbps    0              3.99 MBytes

3.0 - 4.0      157.30 Mbps    0              3.99 MBytes

4.0 - 5.0      157.28 Mbps    0              3.99 MBytes

5.0 - 6.0      157.28 Mbps    0              3.99 MBytes

6.0 - 7.0      157.29 Mbps    0              3.99 MBytes

7.0 - 8.0      157.29 Mbps    0              3.99 MBytes

8.0 - 9.0      157.29 Mbps    0              3.99 MBytes

9.0 - 10.0     157.28 Mbps    0              3.99 MBytes

10.0 - 11.0    157.29 Mbps    0              3.99 MBytes

11.0 - 12.0    157.29 Mbps    0              3.99 MBytes

12.0 - 13.0    157.30 Mbps    0              3.99 MBytes

13.0 - 14.0    157.27 Mbps    0              3.99 MBytes

14.0 - 15.0    157.29 Mbps    0              3.99 MBytes

15.0 - 16.0    157.28 Mbps    0              3.99 MBytes

16.0 - 17.0    157.28 Mbps    0              3.99 MBytes

17.0 - 18.0    157.28 Mbps    0              3.99 MBytes

18.0 - 19.0    157.28 Mbps    0              3.99 MBytes

19.0 - 20.0    157.29 Mbps    0              3.99 MBytes

20.0 - 21.0    157.28 Mbps    0              3.99 MBytes

21.0 - 22.0    157.28 Mbps    0              3.99 MBytes

22.0 - 23.0    157.31 Mbps    0              3.99 MBytes

23.0 - 24.0    157.26 Mbps    0              3.99 MBytes

24.0 - 25.0    167.81 Mbps    0              3.99 MBytes

25.0 - 26.0    157.28 Mbps    0              3.99 MBytes

26.0 - 27.0    157.28 Mbps    0              3.99 MBytes

27.0 - 28.0    157.29 Mbps    0              3.99 MBytes

28.0 - 29.0    157.29 Mbps    0              3.99 MBytes

29.0 - 30.0    157.28 Mbps    0              3.99 MBytes

 

Summary

Interval       Throughput     Retransmits    Receiver Throughput

0.0 - 30.0     152.10 Mbps    0              151.17 Mbps

 

No further runs scheduled.