perfSONAR User Q&A and Other Discussion

[Johann Hugo <>]

I'm making progress, but psconfig tasks = 0, with no mesh grids in Grafana.

I managed to get rid of the "Forbidden for url" messages in psconfig-grafana-agent.log by updating the "grafana-password" field in /etc/perfsonar/psconfig/grafana-agent.json with my new grafana password.

2025-04-17 10:26:10 INFO pid=20827 prog=<module> line=93 msg=Starting pSConfig Grafana agent
2025-04-17 10:26:10 INFO pid=20827 prog=<module> line=104 guid=64885230-33a5-4cbe-9c24-e8c4a6584780 msg=Running agent...
2025-04-17 10:26:10 INFO pid=20827 prog=resolve_address_multi line=110 msg=v6 lookup failed. Cannot resolve address perf-dash.sanren.ac.za. Error Message - The DNS response does not contain an answer to the question: perf-dash.sanren.ac.za. IN AAAA
2025-04-17 10:26:10 INFO pid=20827 prog=reverse_dns_multi line=138 msg=dns reverse lookup failed for fe80::250:56ff:fe9e:cabf%ens160. Error Message - Text input is malformed.
2025-04-17 10:26:10 INFO pid=20827 prog=<module> line=106 guid=64885230-33a5-4cbe-9c24-e8c4a6584780 msg=Agent completed running

I can see a lot of messages in /var/log/perfsonar/psconfig-grafana-agent-transactions.log (attached)

Nothing in /var/log/perfsonar/psconfig-hostmetrics-agent.log

Nothing in /var/log/perfsonar/psconfig-grafana-agent-tasks.log

I now have Grafana entries under perfSONAR pSConfig, but no mesh grids.

Should psconfig show some tasks ?

psconfig stats grafana
Agent Last Run Start Time: 2025-04-17 10:50:37
Agent Last Run End Time: 2025-04-17 10:50:38
Agent Last Run Process ID (PID): 21156
Agent Last Run Log GUID: 522f42bf-7c39-4c77-9e43-5b43acd67221
Total tasks managed by agent: 0

psconfig published
[sanren_100g_servers.json]
    Local File: /usr/lib/perfsonar/web-psconfig/sanren_100g_servers.json
    Web URL: https://perf-dash.sanren.ac.za/psconfig/sanren_100g_servers.json

psconfig remote list
=== Grafana Agent ===
[
   {
      "url": "https://perf-dash.sanren.ac.za/psconfig/sanren_100g_servers.json"
   }
]

=== HostMetrics Agent ===
[
   {
      "url": "https://perf-dash.sanren.ac.za/psconfig/sanren_100g_servers.json"
   }
]

Johann

On Wed, Apr 16, 2025 at 9:17 AM Johann Hugo <> wrote:

Hi Andrew

I also did this at some stage:

 grafana-cli admin reset-admin-password admin

Did this reset the password used by psconfig-grafana-agent ?

Regards

Johann

On Wed, Apr 16, 2025 at 8:34 AM Johann Hugo <> wrote:

Hi Andrew

grafana-agent.json (attached) looks fine. Do psconfig-grafana-agent require authentication credentials or a token to create the Grafana datasource? I've changed the default Grafana password, can this break anything? How can I verify if this is ok ?

Regards

Johann

On Wed, Apr 16, 2025 at 12:13 AM Andrew Lake <> wrote:

Hi Johann,

That error usually means it can’t an HTTP archiver, but looks like you have one of those. In /etc/perfsonar/psconfig/grafana-agent.json do you have any of the following:

-  Is "grafana-datasource-create” set to false? That would cause the issue you see below. If it is not set at all then it will default to true and you are fine.

- Are any of the “datasource_selector” settings under the “display” settings set to “manual” instead of "auto”? That could also cause this.

Otherwise at first glance it looks good.

Thanks,

Andy

On April 15, 2025 at 12:35:41 PM, Johann Hugo () wrote:

Hi all

psconfig-grafana-agent do not generate any grafana dashboards from my psconfig file.

psconfig remote list
=== Grafana Agent ===
[
   {
      "url": "http://perf-pwa.sanren.ac.za/pub/config/sanren_100g_servers.json?format=psconfig"
   }
]

=== HostMetrics Agent ===
[
   {
      "url": "http://perf-pwa.sanren.ac.za/pub/config/sanren_100g_servers.json?format=psconfig"
   }
]

psconfig stats grafana

Agent Last Run Start Time: 2025-04-15 15:13:03
Agent Last Run End Time: 2025-04-15 15:13:04
Agent Last Run Process ID (PID): 80884
Agent Last Run Log GUID: 2dec2519-4c0f-45b7-b449-45d380436824
Total tasks managed by agent: 0

All the testpoints in the mesh are reading the psconfig file and they are submitting results to opensearch. I can see test results in Grafana

https://perf-dash.sanren.ac.za/grafana

This is the errors I get in psconfig-grafana-agent.log

2025-04-15 15:13:04 WARNING pid=80884 prog=_gf_set_home_dashboard line=713 guid=2dec2519-4c0f-45b7-b449-45d380436824 pscheduler_assist_url=https://localhost/pscheduler msg=Unable to set home dashboard: HTTP Error - 403 Client Error: Forbidden for url: https://localhost/grafana/api/org/preferences
2025-04-15 15:13:04 ERROR pid=80884 prog=_gf_create_datasource line=589 guid=2dec2519-4c0f-45b7-b449-45d380436824 config_src=remote config_url=http://perf-pwa.sanren.ac.za/pub/config/sanren_100g_servers.json?format=psconfig msg=Unable to create datasource for https://perf-dash.sanren.ac.za/opensearch: HTTP Error - 403 Client Error: Forbidden for url: https://localhost/grafana/api/datasources
2025-04-15 15:13:04 ERROR pid=80884 prog=_select_gf_datasource line=678 guid=2dec2519-4c0f-45b7-b449-45d380436824 config_src=remote config_url=http://perf-pwa.sanren.ac.za/pub/config/sanren_100g_servers.json?format=psconfig msg=Unable to find a suitable archive to use as Grafana datasource
2025-04-15 15:13:04 ERROR pid=80884 prog=_gf_create_datasource line=589 guid=2dec2519-4c0f-45b7-b449-45d380436824 config_src=remote config_url=http://perf-pwa.sanren.ac.za/pub/config/sanren_100g_servers.json?format=psconfig msg=Unable to create datasource for https://perf-dash.sanren.ac.za/opensearch: HTTP Error - 403 Client Error: Forbidden for url: https://localhost/grafana/api/datasources
2025-04-15 15:13:04 ERROR pid=80884 prog=_select_gf_datasource line=678 guid=2dec2519-4c0f-45b7-b449-45d380436824 config_src=remote config_url=http://perf-pwa.sanren.ac.za/pub/config/sanren_100g_servers.json?format=psconfig msg=Unable to find a suitable archive to use as Grafana datasource
2025-04-15 15:13:04 INFO pid=80884 prog=<module> line=106 guid=2dec2519-4c0f-45b7-b449-45d380436824 msg=Agent completed running

Is this a password or token related issue? Any ideas/pointers ?

Regards

Johann

--
To unsubscribe from this list: https://lists.internet2.edu/sympa/signoff/perfsonar-user

Attachment: psconfig-grafana-agent-transactions.log
Description: Binary data