Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived


Chronological Thread 
  • From: Tao Zhang <>
  • To: Mark Feit <>, "" <>
  • Subject: Re: [perfsonar-user] Perfsonar (automatically) upgraded to 5.0 but testing data not archived
  • Date: Thu, 27 Apr 2023 18:30:19 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=canarie.ca; dmarc=pass action=none header.from=canarie.ca; dkim=pass header.d=canarie.ca; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fTCfkNU9GcaqePNjLg2rKPPnD9pP9SZ3jfE1OKG63UI=; b=OVbNjSrU2S65j6mYNDwQ3OLDSg6SOGQPvu91q2BnGleq3fx0Rhs/xkFdkqKGNIPgRCX3ktj6ebxAzjtdS226ML7AHXAaeUcsURw11bzvHlrcOUGSNLwPu0gU0DOW4vv/MYN2zCcz/KzlxRjyn05UNw83VtArr/Uq+4kcsgx6c/L4eGufiFsDUBjaHEjuNB1LOU+t6Yo+Z2yrt4wkunxQfDBITZ2FaPmdkoFN+Yk35DyR/8Lem4GBJ8ghrPyPUQRx/1ptN6Yc2PdMF68Y0F8ORj22VG5/koRbImKg0x/GWBASDieyI2LloFeWMxJUiemWNDyYDalnbuTaXx1sZDDpow==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Fby2Ir+YbOzz0cggY/1poorGAgipMoOgxLKFTEdOm9joysbQolyfn+5airDimplK6O0GDe47ZVoydvQHXAP8FFHfNuhJPxBufkoHYfgXP0vgd0R9Lez2XLPKX6vXmyfKs1QD+XM73YZwaq3nlG3oZ5r99siwITA/hkTMauEvs95bmL+mOq88xP05wGf1GwXt6mCLkUU1aTcfBpNOB+rZLJrf1U3lR2bB+PiV3pEiTtPA6fYXtUo7C6THe50daQHYG4GQxu1x7/nEKFh9auTg60vaGC1wv0B0tnKOcKID+v1bFQKQxcGyR1y4+9kmpqJCaoFCPUVroPzpd1Qxod/mcw==
  • Msip_labels:

Hi Mark,
The testing data is still not archived to LogStash and OpenSearch. Not sure which part went wrong.
Here is the configuration:
[root@lat-sask1pfs1 tzhang]# cat /etc/perfsonar/psconfig/archives.d/http_logstash.json
{
    "archiver": "http",
    "data": {
        "schema": 2,
        "_url": "https://{% scheduled_by_address %}/logstash",
        "op": "put",
        "_headers": {
            "x-ps-observer": "{% scheduled_by_address %}",
            "content-type": "application/json",
            "Authorization":"Basic cGVyZnNvbmPyOmZmZkt1dGptdzF3SGhqOU5walTT"
        }
    }

The LogStash and OpenSearch services are up.

[root@lat-sask1pfs1 tzhang]# service opensearch.service status
Redirecting to /bin/systemctl status opensearch.service
● opensearch.service - OpenSearch
   Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-04-21 10:42:18 CST; 6 days ago
[root@lat-sask1pfs1 tzhang]# service logstash status
Redirecting to /bin/systemctl status logstash.service
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-04-27 12:21:19 CST; 18s ago

[root@lat-sask1pfs1 tzhang]# cat /var/log/pscheduler/pscheduler.log
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <html><head>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <html><head>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <title>503 Service Unavailable</title>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <title>503 Service Unavailable</title>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  </head><body>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <h1>Service Unavailable</h1>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <p>The server is temporarily unable to service your
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  request due to maintenance downtime or capacity
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  </head><body>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <h1>Service Unavailable</h1>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  <p>The server is temporarily unable to service your
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  request due to maintenance downtime or capacity
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  problems. Please try again later.</p>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  </body></html>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  problems. Please try again later.</p>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  </body></html>
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  121232484: Gave up archiving https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/10931a73-ce1b-4033-9070-7bc3eab1bbb3/runs/eea47b71-7ebf-46fc-b95c-c629edfdf7bb to http
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  121232484: Gave up archiving https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/10931a73-ce1b-4033-9070-7bc3eab1bbb3/runs/eea47b71-7ebf-46fc-b95c-c629edfdf7bb to http
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  121232485: Failed to archive https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/3720e1cc-da4b-417f-a8f3-9be209195cb7/runs/30ff8da1-8d6b-4d95-b0a1-7da37c21e0ab to http: Failed to put result: 503: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Apr 27 12:13:49 lat-sask1pfs1 journal: archiver WARNING  121232485: Failed to archive https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/3720e1cc-da4b-417f-a8f3-9be209195cb7/runs/30ff8da1-8d6b-4d95-b0a1-7da37c21e0ab to http: Failed to put result: 503: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
.....


Thanks,
Tao

From: Mark Feit <>
Sent: Friday, April 21, 2023 4:52 PM
To: Tao Zhang <>; <>
Subject: Re: Perfsonar (automatically) upgraded to 5.0 but testing data not archived
 

External This email originated from outside the organization. Use caution when following links as they could open malicious web sites.



Tao Zhang writes:

 

All our perfSONAR boxes have been upgraded to 5.0, however, the archive shows empty. 

I checked the pscheduler log. The log filled with something like this:

Apr 21 12:27:31 lat-sask1pfs1 journal: archiver WARNING  120686191: Failed to archive https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/b4843fdf-6d63-4a72-a29e-770134ccf853/runs/b7ed7962-377d-4842-90da-952c3e3670ea to esmond: Archiver permanently abandoned registering test after 1 attempt(s): 405: Invalid JSON returned

Apr 21 12:27:34 lat-sask1pfs1 journal: archiver WARNING  <title>503 Service Unavailable</title>

 

5.0 replaced Esmond with LogStash and OpenSearch.  There is a backward-compatibility layer for Esmond, but it is read-only.  If your tasks are archiving to the local system, they will need to use the HTTP archiver and post it to LogStash instead.  These two sections of the documentation should get you there:

 

 

 

Apr 21 12:27:30 lat-sask1pfs1 journal: runner ERROR      File "/usr/libexec/pscheduler/classes/tool/traceroute/run", line 239, in <module>

Apr 21 12:27:30 lat-sask1pfs1 journal: runner ERROR        run_timeout += send_wait * hops

Apr 21 12:27:30 lat-sask1pfs1 journal: runner ERROR    TypeError: unsupported operand type(s) for +=: 'datetime.timedelta' and 'str'

 

This is a bug in pScheduler that appears to have been introduced in the last release.  I’ve already corrected it and added it to the pipeline for the next bugfix release, which should happen sometime during the week of May 1.

 

Based on what I could pull out of the API on your system, these are the tasks that are failing:

 

2023-04-21T14:27:24-06:00 - 2023-04-21T14:30:32-06:00 (Failed)

trace --dest lat-gp-mn.nbren.ca --length 40 --probe-type udp --source lat-sask1pfs1.network.canarie.ca --source-node lat-sask1pfs1.network.canarie.ca --hops 30 --wait PT30S --sendwait PT5S (Run with tool 'traceroute')

https://lat-sask1pfs1.network.canarie.ca/pscheduler/tasks/1aa55201-d3dd-49b4-bda6-99524115462e/runs/0c7a63c4-70bc-4eeb-8181-dfcb4195d148

 

As a stopgap, removing the ‘sendwait’ option from your trace tasks won’t trip the faulty code.  Note that this will alter the character of the measurement.

 

--Mark

 




Archive powered by MHonArc 2.6.24.

Top of Page