perfSONAR User Q&A and Other Discussion

[Johann Hugo <>]

Hi all

MadDash not showing any test results. it's configured to read results from both esmond and opensearch. This is in our remote psconfig template

  "archives":{
    "esmond_central":{
      "archiver":"esmond",
      "data":{
        "measurement-agent":"{% scheduled_by_address %}",
        "url":"https://perf-cm.sanren.ac.za/esmond/perfsonar/archive/"
      }
    },  
    "opensearch_central":{
      "archiver": "http",
      "data": {
        "schema": 3,
        "_url": "https://perf-cm.sanren.ac.za/logstash",
        "verify-ssl": false,
        "op": "put",
        "_headers": {
          "x-ps-observer": "{% scheduled_by_address %}",
          "content-type": "application/json"
        }
      }
    }
  },

psconfig validate http://pwa-test.sanren.ac.za/sanren_mesh_2023.json
Loading template ...... OK
Validating JSON schema ...... OK
Verifying object references ...... OK
pSConfig JSON is valid

psconfig maddash-stats
Agent Last Run Start Time: 2023/05/22 15:42:30
Agent Last Run End Time: 2023/05/22 15:45:45
Agent Last Run Process ID (PID): 1881
Agent Last Run Log GUID: 7FBE38F0-F8A6-11ED-988C-506B81FE1034
Total grids managed by agent: 12
From remote definitions: 12
    http://pwa-test.sanren.ac.za/sanren_mesh_2023.json: 12

perfSONAR nodes are posting pscheduler test results to both esmond and logstash:
[root@perf-cm ~]# tail -F /var/log/httpd/ssl_access_log
155.232.40.173 - - [22/May/2023:16:26:34 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.40.173 - - [22/May/2023:16:26:34 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.173 - - [22/May/2023:16:26:34 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.173 - - [22/May/2023:16:26:34 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.40.26 - - [22/May/2023:16:26:35 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [22/May/2023:16:26:35 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.50.1 - - [22/May/2023:16:26:36 +0200] "PUT /logstash HTTP/1.1" 403 210
155.232.50.1 - - [22/May/2023:16:26:36 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.40.26 - - [22/May/2023:16:26:37 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.40.26 - - [22/May/2023:16:26:37 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.90 - - [22/May/2023:16:26:38 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.90 - - [22/May/2023:16:26:38 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
192.33.10.253 - - [22/May/2023:16:26:38 +0200] "PUT /logstash HTTP/1.1" 403 210
192.33.10.253 - - [22/May/2023:16:26:38 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.40.90 - - [22/May/2023:16:26:39 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.90 - - [22/May/2023:16:26:39 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.40.146 - - [22/May/2023:16:26:41 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.146 - - [22/May/2023:16:26:41 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.50.1 - - [22/May/2023:16:26:41 +0200] "PUT /logstash HTTP/1.1" 403 210
155.232.50.1 - - [22/May/2023:16:26:41 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299
155.232.40.177 - - [22/May/2023:16:26:41 +0200] "POST /esmond/perfsonar/archive/? HTTP/1.1" 503 299

tail -F /var/log/httpd/ssl_error_log
[Mon May 22 17:54:43.007573 2023] [proxy_http:error] [pid 12211] [client 155.232.40.173:51362] AH01114: HTTP: failed to make connection to backend: localhost
[Mon May 22 17:54:44.343206 2023] [proxy:error] [pid 12083] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:5000 (localhost) failed

Logstash is listing for data on https://perf-cm.sanren.ac.za/logstash

Esmond is not wating for data on https://perf-cm.sanren.ac.za/esmond/perfsonar/archive/

Esmond is listening on port 9160, no django errors.
2023-05-12 00:30:07,699 [INFO] /usr/lib/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

When I connect to the URL from my pc I get - Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again late

No postgresql errors:
May 22 16:19:35 perf-cm.sanren.ac.za postmaster[1354]: 2023-05-22 16:19:35.160 SAST [1354] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"

I recreated the SQL database +
/usr/pgsql-10/bin/postgresql-10-setup initdb
/usr/lib/perfsonar/scripts/system_environment/configure_esmond --force

/usr/sbin/esmond_manage list_user_ip_address
esmond [INFO] Checking/creating column families
esmond [INFO] Schema check done
esmond [DEBUG] Opening ConnectionPool
esmond [INFO] Connected to ['localhost:9160']
sanren 192.33.10.253/32
sanren 192.33.10.254/32
sanren 155.232.40.0/24
sanren 155.232.195.0/24
sanren 155.232.21.202/32
sanren 155.232.50.0/24
sanren 170.39.8.0/24
sanren 146.64.80.0/24

Any ideas ?

Regards

Johann

--

SANReN Engineer

South African National Research Network (SANReN)

National Integrated Cyber Infrastructure System (NICIS)

CSIR NextGen Enterprises and Institutions Cluster

Office: 012 841 2066Email: , Website: www.sanren.ac.za / www.csir.co.za