perfsonar-user - [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived

Subject: perfSONAR User Q&A and Other Discussion

List archive

[perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived

From: Tao Zhang <>
To: "" <>
Subject: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived
Date: Fri, 21 Apr 2023 18:43:56 +0000
Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=canarie.ca; dmarc=pass action=none header.from=canarie.ca; dkim=pass header.d=canarie.ca; arc=none
Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aCdUXb/dYeC8gKkJOdGRk6EDQMTCJdCCmqBZowkYvHQ=; b=GdrGq/MjrollalXDloiSC33dBb6PJPXxAIh0pd6XkSXlZRP/n/cNsd09Kt9xYPPjOprtU4pMKWk/9kZyprly03NOGtCn+rNG/oCwfNawx4Xw11kZAhhGUsFNR40O3xszGsXmAKWUOzZEHp5ATdrPGvP9cJRc+b/Om5f25D7FcFu2qXyIIJgZYjsC6oBql4zYE2OPmr/W6yjnFbTRfrQ7g8dVzf87EI00cAIBJqE+tSXAfvcqRdSYdrsx19qEk5pNIUAyS5xh3oxYMfsoDYA+Xmt2nXw3n/19fE+XUJafBZNC0xFeMzfMotgRpzj0Px+LtkhXzNhGXgP/ALBAiLv5hA==
Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mz7J8KMIjyFRqdZWtmM6MJFFE0HknnRjDYFAMqL1zSnjzzmG6mVgTFHMvbpJ6POcbE5H6Jm0cU9qU7y7OFl8bO8ysT8DdYRoL0Umfz+fBsOZACNV1xvH2YT0O+A+huT0nvZDL/T39S/pW9LYzhQTYMOZwiAmJFFwiYLtTxR5/moFvRewwt8C+mf2onkmUmfYBgIwqUNlC3nMGmcPW7V8RAlGOtQYthU+oDHXHlj96xg6SZgAqeZE6APJ1SBruP8/GJfyzrxR1BsFRJPi4iptsX4TRe4YOVjkCodmudqij/QTmFPQ34p2MVZjc153l5UHCe49rKmuVIGrokFY9q6KGQ==
Msip_labels:

Hi team,

All our perfSONAR boxes have been upgraded to 5.0, however, the archive shows empty.

Here are some commands output for one of the boxes:

[root@lat-sask1pfs1 tzhang]# pscheduler troubleshoot

Performing basic troubleshooting of lat-sask1pfs1.network.canarie.ca.

lat-sask1pfs1.network.canarie.ca:

Measuring MTU... 65535 (Local)

Looking for pScheduler... OK.

Fetching API level... 5

Checking clock... OK.

Exercising API... Archivers... Contexts... Tests... Tools... OK.

Fetching service status... OK.

Checking services... Ticker... Scheduler... Runner... Archiver... OK.

Checking limits... OK.

Idle test.... 8 seconds... Finished... Checking archiving... OK.

pScheduler appears to be functioning normally.

[root@lat-sask1pfs1 tzhang]# psconfig remote list

=== pScheduler Agent ===

[

{

"url" : "https://lat-otwa3pfs1.network.canarie.ca/canarieTest.json"

}

]

[root@lat-sask1pfs1 tzhang]# psconfig validate https://lat-otwa3pfs1.network.canarie.ca/canarieTest.json

Loading template ...... OK

Validating JSON schema ...... OK

Verifying object references ...... OK

pScheduler Validation (Quick) ...... OK

pSConfig JSON is valid

[root@lat-sask1pfs1 tzhang]# psconfig pscheduler-stats

Agent Last Run Start Time: 2023/04/21 12:00:34

Agent Last Run End Time: 2023/04/21 12:13:53

Agent Last Run Process ID (PID): 3011

Agent Last Run Log GUID: 69B22F3A-E06E-11ED-B681-84FE45962F80

Total tasks managed by agent: 82

From remote definitions: 82

https://lat-otwa3pfs1.network.canarie.ca/canarieTest.json: 82

However, the archive link shows empty

https://lat-sask1pfs1.network.canarie.ca/esmond/perfsonar/archive/

I checked the pscheduler log. The log filled with something like this:

Apr 21 12:27:31 lat-sask1pfs1 journal: archiver WARNING 120686192: Gave up archiving https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/b4843fdf-6d63-4a72-a29e-770134ccf853/runs/b7ed7962-377d-4842-90da-952c3e3670ea to http

Apr 21 12:27:31 lat-sask1pfs1 journal: archiver WARNING 120686191: Failed to archive https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/b4843fdf-6d63-4a72-a29e-770134ccf853/runs/b7ed7962-377d-4842-90da-952c3e3670ea to esmond: Archiver permanently abandoned registering test after 1 attempt(s): 405: Invalid JSON returned

.....

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING <html><head>

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING <title>503 Service Unavailable</title>

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING </head><body>

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING <h1>Service Unavailable</h1>

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING <p>The server is temporarily unable to service your

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING request due to maintenance downtime or capacity

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING problems. Please try again later.</p>

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING </body></html>

...

Apr 21 12:27:30 lat-sask1pfs1 journal: runner ERROR 121552232: Run failed 1: Traceback (most recent call last):

Apr 21 12:27:30 lat-sask1pfs1 journal: runner ERROR File "/usr/libexec/pscheduler/classes/tool/traceroute/run", line 239, in <module>

Apr 21 12:27:30 lat-sask1pfs1 journal: runner ERROR run_timeout += send_wait * hops

Apr 21 12:27:30 lat-sask1pfs1 journal: runner ERROR TypeError: unsupported operand type(s) for +=: 'datetime.timedelta' and 'str'

I’m not sure what to do now. Any suggestions?

Thanks,

Tao

[perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Tao Zhang, 04/21/2023
- Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Mark Feit, 04/21/2023
  - Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Tao Zhang, 04/27/2023
    - Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Zachary Newell, 04/27/2023
      - Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Tao Zhang, 04/27/2023
        
        Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Zachary Newell, 04/27/2023
        
        Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Tao Zhang, 04/28/2023
        
        Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Mark Feit, 04/28/2023
        
        Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived, Zachary Newell, 04/28/2023

List archive

[perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived