perfSONAR User Q&A and Other Discussion

[Johann Hugo <>]

Thanks

logstash test results from the testpoints are now reaching the archive/maddash server, but the grid is still orange

tail -F /var/log/httpd/ssl_access_log | grep logstash
155.232.40.26 - - [17/May/2023:20:16:41 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.2 - - [17/May/2023:20:16:43 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.2 - - [17/May/2023:20:16:44 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.2 - - [17/May/2023:20:16:44 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.203 - - [17/May/2023:20:16:44 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [17/May/2023:20:16:45 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [17/May/2023:20:16:51 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [17/May/2023:20:16:55 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [17/May/2023:20:16:56 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [17/May/2023:20:17:00 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.203 - - [17/May/2023:20:17:01 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [17/May/2023:20:17:01 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.2 - - [17/May/2023:20:17:03 +0200] "PUT /logstash HTTP/1.1" 200 2
155.232.40.26 - - [17/May/2023:20:17:03 +0200] "PUT /logstash HTTP/1.1" 200 2

Do I need to add some json to the maddash server as well ?

What log files should I look at next on the archive/maddash server.

tail -F /var/log/logstash/logstash-deprecation.log
[2023-05-17T20:21:26,007][WARN ][deprecation.logstash.codecs.json][pscheduler][5722c59573404fd6e5c3cf82c33b85942f413950c8644fc62db757dae6eedbc8] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode

tail -F /var/log/logstash/logstash-plain.log
[2023-05-17T16:32:32,737][WARN ][org.logstash.dissect.Dissector][pscheduler][e27ecd5933979d3c114ddc9b45a0b1a0c51928e9a849bb598bd38e0cee61e857] Dissector mapping, field not found in event {"field"=>"[pscheduler][task_href]", "event"=>{"headers"=>{"x_forwarded_host"=>"155.232.195.55", "http_accept"=>"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "upgrade_insecure_requests"=>"1", "content_length"=>"0", "request_method"=>"GET", "http_host"=>"155.232.195.55", "accept_language"=>"en-GB,en-US;q=0.9,en;q=0.8", "request_path"=>"/", "http_version"=>"HTTP/1.1", "http_user_agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36", "accept_encoding"=>"gzip, deflate", "x_forwarded_server"=>"perf-cm.sanren.ac.za", "x_forwarded_for"=>"146.64.80.50", "connection"=>"Keep-Alive"}, "host"=>"127.0.0.1", "@timestamp"=>2023-05-17T14:32:32.635Z, "message"=>"", "@version"=>"1"}}
[2023-05-17T16:32:32,741][WARN ][org.logstash.dissect.Dissector][pscheduler][e27ecd5933979d3c114ddc9b45a0b1a0c51928e9a849bb598bd38e0cee61e857] Dissector mapping, field not found in event {"field"=>"[pscheduler][run_href]", "event"=>{"headers"=>{"x_forwarded_host"=>"155.232.195.55", "http_accept"=>"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "upgrade_insecure_requests"=>"1", "content_length"=>"0", "request_method"=>"GET", "http_host"=>"155.232.195.55", "accept_language"=>"en-GB,en-US;q=0.9,en;q=0.8", "request_path"=>"/", "http_version"=>"HTTP/1.1", "http_user_agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36", "accept_encoding"=>"gzip, deflate", "x_forwarded_server"=>"perf-cm.sanren.ac.za", "x_forwarded_for"=>"146.64.80.50", "connection"=>"Keep-Alive"}, "host"=>"127.0.0.1", "@timestamp"=>2023-05-17T14:32:32.635Z, "message"=>"", "@version"=>"1"}}

Regards

Johann

On Wed, May 17, 2023 at 5:02 PM Andrew Lake <> wrote:

Hi Johann,

Looks like our docs and the helper script are putting that “_meta” field in the wrong place, sorry about that. Move _meta outside the “data” element and you should be good to go. Example:

{

    "archiver": "http",

    "data": {

        "schema": 2,

        "_url": "https://{% scheduled_by_address %}/logstash",

        "verify-ssl": false,

        "op": "put",

        "_headers": {

            "x-ps-observer": "{% scheduled_by_address %}",

            "content-type": "application/json", "Authorization":"Basic cGVyZnNvbmFyOmRMRE50dmt0Q253elZNdXRlbUdr"

        },

        "schema": 3,

        "_url": "https://perf-cm.sanren.ac.za/logstash",

        "verify-ssl": false,

        "op": "put",

        "_headers": {

            "x-ps-observer": "{% scheduled_by_address %}",

            "content-type": "application/json"

        }

    },

    "_meta": {

        "esmond_url": "https://perf-cm.sanren.ac.za/esmond/perfsonar/archive/"

    }

}

Thanks,

Andy

On May 17, 2023 at 3:29:02 AM, Johann Hugo () wrote:

Looks like pscheduler keeps on running (pscheduler monitor shows green tests) when I merge the json into /etc/perfsonar/psconfig/archives.d/http_logstash.json

{
    "archiver": "http",
    "data": {
        "schema": 2,
        "_url": "https://{% scheduled_by_address %}/logstash",
        "verify-ssl": false,
        "op": "put",
        "_headers": {
            "x-ps-observer": "{% scheduled_by_address %}",
            "content-type": "application/json", "Authorization":"Basic cGVyZnNvbmFyOmRMRE50dmt0Q253elZNdXRlbUdr"
        },
        "schema": 3,
        "_url": "https://perf-cm.sanren.ac.za/logstash",
        "verify-ssl": false,
        "op": "put",
        "_headers": {
            "x-ps-observer": "{% scheduled_by_address %}",
            "content-type": "application/json"
        },
        "_meta": {
            "esmond_url": "https://perf-cm.sanren.ac.za/esmond/perfsonar/archive/"
        }
    }
}

but psconfig-pscheduler-agent is complaining about invalid archiver data. 

tail -F  /var/log/perfsonar/psconfig-pscheduler-agent.log

2023/05/17 09:18:50 WARN pid=4074 prog=perfSONAR_PS::PSConfig::PScheduler::Agent::_run_end line=229 guid=F101A9E8-F482-11ED-B6A0-C740E2E80815 msg=Problem adding test latencybg(ps-8-pta-unisa-10g.perfsonar.ac.za->ps-1-pta-csir-10g.perfsonar.ac.za), continuing with rest of config: BAD REQUEST: Invalid archiver data: At /: Additional properties are not allowed ('_meta' was unexpected)
2023/05/17 09:18:50 WARN pid=4074 prog=perfSONAR_PS::PSConfig::PScheduler::Agent::_run_end line=229 guid=F101A9E8-F482-11ED-B6A0-C740E2E80815 msg=Problem adding test latencybg(ps-8-pta-unisa-10g.perfsonar.ac.za->ps-12-hla-sac-10g.perfsonar.ac.za), continuing with rest of config: BAD REQUEST: Invalid archiver data: At /: Additional properties are not allowed ('_meta' was unexpected)
2023/05/17 09:18:50 WARN pid=4074 prog=perfSONAR_PS::PSConfig::PScheduler::Agent::_run_end line=229 guid=F101A9E8-F482-11ED-B6A0-C740E2E80815 msg=Problem adding test latencybg(ps-8-pta-unisa-10g.perfsonar.ac.za->ps-6-jnb-wits-10g.perfsonar.ac.za), continuing with rest of config: BAD REQUEST: Invalid archiver data: At /: Additional properties are not allowed ('_meta' was unexpected)

Test results are not posted to local or central logstash archives + errors for the local esmond archives. Pscheduler is complaining about Invalid JSON returned.

/var/log/pscheduler/pscheduler.log

May 17 08:09:23 ps-8-pta-unisa journal: archiver WARNING  43552228: Gave up archiving https://ps-8-pta-unisa/pscheduler/tasks/0e843dcb-dfcf-4df9-902e-3082ff3a5c2a/runs/1ae3c79a-7600-41c8-8699-d62b8d271db7 to esmond
May 17 08:09:23 ps-8-pta-unisa journal: archiver WARNING  43552228: Gave up archiving https://ps-8-pta-unisa/pscheduler/tasks/0e843dcb-dfcf-4df9-902e-3082ff3a5c2a/runs/1ae3c79a-7600-41c8-8699-d62b8d271db7 to esmond
May 17 08:09:23 ps-8-pta-unisa journal: archiver WARNING  43552227: Failed to archive https://ps-8-pta-unisa/pscheduler/tasks/0e843dcb-dfcf-4df9-902e-3082ff3a5c2a/runs/1ae3c79a-7600-41c8-8699-d62b8d271db7 to esmond: Archiver permanently abandoned registering test after 1 attempt(s): 503: Invalid JSON returned

Any ideas ?

Regards

Johann

On Mon, May 15, 2023 at 4:10 PM Johann Hugo <> wrote:

I forgot this part

tail -F  /var/log/pscheduler/pscheduler.log

May 15 16:09:18 ps-2-dur-dut journal: archiver WARNING  4793372: Failed to archive https://ps-2-dur-dut/pscheduler/tasks/fb0fe1df-a6cf-4d20-942c-9a25640a97dd/runs/5086d0ac-7b6d-4a0e-a1b0-9c210a7cbdde to esmond: Archiver permanently abandoned registering test after 1 attempt(s): 503: Invalid JSON returned

On Mon, May 15, 2023 at 3:44 PM Johann Hugo <> wrote:

Hi Andy

I've tried both methods on one of my testpoints, but no luck.

1) Adding raw json to my PWA config:

 The testpoint downloads the config. "psconfig pscheduler-stats list" shows all the tests that it should run. I can see the tests under "pscheduler monitor", but all the tests are grey. See attached mesh config file created by PWA.

2) Create a  json archive file under /etc/perfsonar/psconfig/archives.d/

With my old mesh config file the tests under "pscheduler monitor" are green. When I create a json archive file under under /etc/perfsonar/psconfig/archives.d, then the tests under "pscheduler monitor" turn grey. When I delete the json archive file (attached perf-cm.json) it will go green again. 

I've added the IP of my own pc to /etc/httpd/conf.d/apache-logstash.conf on the central archive server and I can see it's waiting for test results when I open https://perf-cm.sanren.ac.za/logstash, but if I tail -F /var/log/httpd/ssl_access_log only esmond results are posted to the server, no logstash results.

Regards

Johann

On Fri, May 12, 2023 at 3:24 PM Andrew Lake <> wrote:

Hi,

You would add that to the PWA archiver config. I don’t have a PWA interface in front of me, but I believe there is a way to add raw json to the archiver config, so you should be able to use that since there will not be pulldowns for all the options (a PWA refresh is a something we are looking at currently, but still quite a ways out). Alternatively, on the measurement hosts (i.e. the host running pscheduler and doing the actual measurements) you can add the JSON from that command into a .json file) under /etc/perfsonar/psconfig/archives.d. You can name the file whatever you want as long as it ends in .json.

Thanks,

Andy

On May 12, 2023 at 4:31:03 AM, Johann Hugo () wrote:

Hi all

I'm trying to get a new central measurement archive running, using authentication by IP address:

https://docs.perfsonar.net/multi_ma_install.html

I've done the following:

- yum install perfsonar-archive

- vi /etc/httpd/conf.d/apache-logstash.conf 

     Require ip 155.232.40.0/24

- /usr/lib/perfsonar/archive/perfsonar-scripts/psconfig_archive.sh -a none -n perf-cm.sanren.ac.za

{
    "archiver": "http",
    "data": {
        "schema": 3,
        "_url": "https://perf-cm.sanren.ac.za/logstash",
        "verify-ssl": false,
        "op": "put",
        "_headers": {
            "x-ps-observer": "{% scheduled_by_address %}",
            "content-type": "application/json"
        },
        "_meta": {
            "esmond_url": "https://perf-cm.sanren.ac.za/esmond/perfsonar/archive/"
        }
    }
}

According to the docs, this is the next step:

Copy above to your central pSConfig template and your measurement hosts should begin archiving

I'm not sure what or how I should do this. Should I add it somewhere to the PWA json config file that the measurement hosts download, or somewhere on my archive/MadDash server ?

Regards

Johann

--

SANReN Engineer

South African National Research Network (SANReN)

National Integrated Cyber Infrastructure System (NICIS)

CSIR NextGen Enterprises and Institutions Cluster

Office: 012 841 2066Email: , Website: www.sanren.ac.za / www.csir.co.za

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user