perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi,

Thanks for reaching out and sorry for the response delay. Comments inline below:

On March 5, 2024 at 9:53:04 AM, Jackson, Wayne P. (GSFC-585.0)[Axient] () wrote:

I have a couple of (most likely stupid) questions:  

First, a quick background:

I have several “Test Points” on separate subnets and one legacy Esmond Archive on its own network behind a firewall. I have now installed and configured a new secondary Archive for PS 5.x, using OpenSearch of course. 

I configure the test points from the “/usr/lib/perfsonar/web-psconfig” path and have one json file configured to send to both Esmond and OpenSearch, where MaDDash reads from Legacy Esmod: Here is the portion of the json where I configured the new “endpsma02 logstash” which is successfully pushed out to one of our test points:

"nsg-opensearch": {

      "archiver": "http",

      "data":{

          "schema":3,

          "_url":https://endpsma02.eosdis.nasa.gov/logstash,

          "verify-ssl":false,

          "op":"put",

          "_headers":{

          "x-ps-observer":"{% scheduled_by_address %}",

          "content-type":"application/json",

          "Authorization":"Basic XXXXXXXXXX.XXXXXX.k94JuLT/ab/S/q1ym8/"

          }

 

Questions:

• In the “Authorization” should this be the value in the “/etc/perfsonar/opensearch/logstash_logn” file and should this include the “perfsonar:” portion?

It actually grabs the value from /etc/perfsonar/logstash/proxy_auth.json which is the credentials you referenced but in base64 encoded form. No “perfsonar:” needed. 

• 
  I have 443 opened between the test point and the new archive server but do I also need additional ports open: 11283 for logstash and or 9200 for OpenSearch? 

You do not need 11283 nor 9200 open. Those ports only listen on localhost, and apache acts as a reverse proxy to those ports. Your testpoints should only be talking to 443, so sounds like you are good if you have opened that. 

• 
  I have a simple python script that uses the key and cert in /etc/opensearch to see the indices and some data in OpenSearch but I do not see an index or docs for any actual data from the new system, which I assume is still blocked or not configured correctly to send result data back to the new Archive server.

• Here are the indices from that query:

 

health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size

green  open   security-auditlog-2024.02.28 gk9JKesIT3qI-yBPJK_OAg   1   0         45            0    187.4kb        187.4kb

green  open   .opensearch-observability    ep_I8W4qQu24G494jVhYEw   1   0          0            0       208b           208b

green  open   security-auditlog-2024.02.14 GBa2-n4GTYqsZvp_sKrw6w   1   0         10            0    124.6kb        124.6kb

green  open   security-auditlog-2024.03.04 pR6FP9y6TpWd1H_S2MgNZw   1   0         23            0     83.9kb         83.9kb

green  open   security-auditlog-2024.02.09 vUR8ySZSTq-yDADI9Ykecw   1   0          6            0     92.1kb         92.1kb

green  open   security-auditlog-2024.02.29 -4VK02a8RgSwb6HUpVqg0Q   1   0         44            0    136.7kb        136.7kb

green  open   security-auditlog-2024.03.01 IZ0dahHCSU25oH2v75jQPg   1   0         25            0    121.8kb        121.8kb

green  open   security-auditlog-2024.02.13 IXeyNPgASfOSUDlA1v7jAQ   1   0          6            0     84.5kb         84.5kb

green  open   .opendistro_security         VAHoPa-8RlOKFZ00ZKKmSQ   1   0         10            0     75.3kb         75.3kb

 

 

As I am completely new to OpenSearch please forgive the most likely stupid questions.

If it helps, we have a script that will ship with the forthcoming 5.1.0 for troubleshooting the archive that you can try running on your box. It will check a bunch of things related to opensearch and logstash (including whether it can see any data). You can run it with the following on your archive box:

curl -s https://raw.githubusercontent.com/perfsonar/archive/5.1.0/perfsonar-archive/perfsonar-archive/bin/commands/troubleshoot | sudo python3

In 5.1.0 this will just be the “pscheduler troubleshoot” command but in my testing it runs just fine on 5.0.X archives. You can open the URL if you are curious what it is checking. If everything looks good, you may want to check your testpoints. Specifically are there any error in /var/log/pscheduler/pscheduler.log complaining about archiving?

Hope that helps,

Andy

 

 

V/R,

Wayne Jackson

Code 423

EOSDIS

(480)540-3279

 

 

 

-- 
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user