perfSONAR User Q&A and Other Discussion

["Robinson, John-Paul" <>]

Just following up with final confirmation on this thread that the error with getting bi-directional tests to archive does appear related to the archiver URL generated during toolkit install.

After getting a working install an test instances documented earlier in the thread, I was able to get my inbound tests (ie. those registered in a remote perfsonar instance against a iperf3 -s run on my instance) to properly post the results so that they show up in my instance UI.

It appears that during the toolkit install the file/etc/perfsonar/psconfig/archives.d/http_logstash.json   has the _url parameter set to "http://localhost/logstash".  This json snippet is included as the archiver target in scheduled tests.  When a remote site attempts to archive to this address to routes to its local interface and will fail.  I fixed this by  simply replacing "localhost" with my fully qualified domain name.  This is on an Ubuntu 22.04 install base.

I had an additional issue on my deployment but I suspect this was a result of the an earlier install that was auto-updated.  Noting this here in case others run into a similar problem.  My archives config stanza did not include an authentication token.  I worked around this by commenting out "Require valid-user" and uncommenting "Require host gov edu" in the logstash apache proxy auth config file apache-logstash.conf.   I plan to fix the config to use token based auth and return to using valid-user.  This was an Alma9 install base.

I've opened and issue. https://github.com/perfsonar/toolkit/issues/486

I'll file a patch if I  figure out the correct templating syntax.

---

From: Robinson, John-Paul <>
Sent: Tuesday, August 13, 2024 11:16 PM
To: <>; Robinson, John-Paul <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

Yes.  Yes there is.  Send kill -s USR1 to the pscheduler runner process.  That will add it's debug log including the start of the commands to the /var/log/pscheduler/pscheduler.log

I'm fairly certain I tried that earlier but couldn't be sure.  I turned off my pscheduler-api  debug logging and then I restarted my test nodes to clear out any stale state. 

After the restart i sent USR1 to the runner and focused only on those debugs.   I was able to do manual throughput tests inbound  and outbound from my test node. I then went to the UI and deleted and re-added a scheduled test against the peer node.  This time I used host names instead of IP addresses (don't think that was issue, just noting).

The debug logs now show both client and server iperf3 processes being started, suggesting  inbound and outbound tests running successfully.

I went to the UI and now I see both the inbound and outbound results.  I want back in time and see that they did just start showing up.  

I'm not convinced any of the above actions had anything to do with fixing this.  In earlier debug I noticed that there were errors posting to the logstash.  Like messages about incorrect credentials.  I opened the logstash config to allow local lan peers to write.    But I also noticed that the /etc/perfsonar/psconfig/archives.d/http_logstash.json had the  url listed as "localhost".   I changed that to the hostname of the local machine.  I think this is what may have fixed it.  I'm wondering if by using localhost the tests were trying to be updated against the remote testers logstash instead of getting sent back to the originator of the test.  I'm not sure how these mechanism work, so i'll need to retest on a clean build.

In any case, bi-directional tests are now logging in my test node and showing up on the web UI.

---

From: <> on behalf of "Robinson, John-Paul" <>
Sent: Tuesday, August 13, 2024 4:40 PM
To: <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

Is there a way to turn on the debug logging in /usr/lib/pscheduler/classes/tool/iperf3/run via a config setting?

---

From: Robinson, John-Paul <>
Sent: Sunday, August 11, 2024 12:02 PM
To: <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

I think I've narrowed down the issue further.  I'm now limiting my configuration to just trying to do a bidirectional test from one of the nodes to the other.  I'm directly workding with the psconfig template created by the web ui.

What appears to be happening is that psconfig correctly defines the bidirectional tests but when they get registered with the lead host they are end up both being  defined in the same direction.

Here's the transaction log that shows the bidirectional tests getting created.  Based on my understanding of the docs,  the lead_url is where the test should be registered.

2024-08-11 10:48:46 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" msg=Creating tasks

2024-08-11 10:48:47 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" checksum=rLl0+VaPhR0XGD63IMRS+g lead_url=https://192.168.1.53/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.93", "dest-node": "192.168.1.93", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.53", "source-node": "192.168.1.53", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-12T15:48:46Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-11 10:48:55 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" checksum=9n9gQQ4gmO+yddcUw6EyOA lead_url=https://192.168.1.93/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.53", "dest-node": "192.168.1.53", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.93", "source-node": "192.168.1.93", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-12T15:48:46Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-11 10:49:02 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" msg=Done creating tasks

When I look at the tests that are scheduled, however, both hosts have a source and destination that are the same, leading to only a unidirectional test.

Here's the schedule from the ps-521-int machine which is configured with the bidirectional test template

ubuntu@ps-521-int:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-11T16:45:25+00:00 - 2024-08-11T16:45:54+00:00 (Scheduling)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521-int/pscheduler/tasks/9e3b8ea9-49f5-4e66-8aee-659e8d0aedee/runs/7c70dd83-4bc7-40af-8c56-a3bac6c231ba

Here's the schedule from the peer node ps-521, note the source and dest are the same as above.

ubuntu@ps-521:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-11T16:45:25+00:00 - 2024-08-11T16:45:54+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521/pscheduler/tasks/9e3b8ea9-49f5-4e66-8aee-659e8d0aedee/runs/7c70dd83-4bc7-40af-8c56-a3bac6c231ba

I'm guessing these are the same tests, but that makes me wonder why i'm not seeing a similar pair for the second task in the agent transaction log above.

Thanks for any guidance.

PS. I found the debug logging feature for the pscheduler-api. 🙂

touch /var/pscheduler-server/pscheduler-api-debug

journalctl -u apache2

I'll let you know if I find anything interesting.

---

From: <> on behalf of "Robinson, John-Paul" <>
Sent: Saturday, August 10, 2024 1:32 PM
To: <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

Hi Folks,

I've done some further investigation.  I cleared out my back log of failed tests and then set up just one of the two nodes to do the bi-directional tests.

On the node 192.168.1.53 on which I re-enabled the bidirectional tests, I see the following in the/var/log/perfsonar/psconfig-pscheduler-agent-transactions.log

2024-08-10 12:59:57 INFO guid=a07f9d62-64cf-46ad-b3e4-389d0d968bcd action="create" checksum=rLl0+VaPhR0XGD63IMRS+g lead_url=https://192.168.1.53/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.93", "dest-node": "192.168.1.93", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.53", "source-node": "192.168.1.53", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-11T17:59:57Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-10 13:00:05 INFO guid=a07f9d62-64cf-46ad-b3e4-389d0d968bcd action="create" checksum=9n9gQQ4gmO+yddcUw6EyOA lead_url=https://192.168.1.93/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.53", "dest-node": "192.168.1.53", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.93", "source-node": "192.168.1.93", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-11T17:59:57Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-10 13:00:13 INFO guid=a07f9d62-64cf-46ad-b3e4-389d0d968bcd action="create" msg=Done creating tasks

This looks like the bi-directional tests get set up.  They should run a throughput once per hour.

When I look at the test schedule on both of the nodes I don't see any bi-directional tests scheduled.  They are all just one directional tests.

Here are the tests on the node on which the tests were configured (192.169.1.53)

ubuntu@ps-521-int:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-10T18:56:53+00:00 - 2024-08-10T18:57:22+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521-int/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/12ee00ea-23a4-42a5-81dd-38b105a0ef12

2024-08-10T19:59:24+00:00 - 2024-08-10T19:59:53+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521-int/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/4fd580bd-c84d-43c4-9331-f7e5a2ad0dfe

Here are the tests on the node which is the target of the tests (192.168.1.93)

ubuntu@ps-521:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-10T18:56:53+00:00 - 2024-08-10T18:57:22+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/12ee00ea-23a4-42a5-81dd-38b105a0ef12

2024-08-10T19:59:24+00:00 - 2024-08-10T19:59:53+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/4fd580bd-c84d-43c4-9331-f7e5a2ad0dfe

As you can see, all the tests are outbound from 192.168.1.53.  I can give a bigger time window but it doesn't show any reverse tests.

I'm not sure where to look next.  Is there another log related to /var/log/perfsonar/psconfig-pscheduler-agent-transactions.log that might be helpful in understanding why the reverse tests appears to be schedule but then doesn't make it into the schedule?

Thanks,

John-Paul

---

From: Robinson, John-Paul <>
Sent: Wednesday, August 7, 2024 12:43 PM
To: <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

Hi,

Thanks for the feedback.  I'm working with perfsonar 5.1.2 so it's the grafana interface.   I navigated to the the pair explorer plot you mentioned.  I see all my hosts I have tests set up in the destination drop-down, but I only see my own hosts in the source.  So I can't swap the tests results around. 

---

From: Szymon Trocha
Sent: Wednesday, August 7, 2024 6:00 AM
To: Robinson, John-Paul
Cc: 
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

W dniu 05.08.2024 o 01:41, "Robinson, John-Paul" (via perfsonar-user Mailing List) pisze:

Hi,

I'm wondering what steps are necessary to enable the plotting of the forward and reverse thoughput results in the web UI.   This may actually be rooted in an issue of those results not being logged or the tests not being performed. In any case, I'd like to fix the problem.

In ps releases prior to 5.x I was able to just set up tests and select "send and receive" in the test set up UI and the results would show up for both directions of successfully completed tests.  In all my setups since switching to 5x, this has not worked out of the box.

I did a fresh install of 5.1.2 on two test boxes on Ubuntu 22.04 on the same lan segment.  I set up a throughput test on each one pointing at the other node.  I selected the "Send and Recieve" in the advance testing parameters.   When I look at the plots I only see the tests that are outbound from the host.  I never see the reverse (inbound) test results. I can run tests in both directions from the command line using pscheduler.

I'm not sure where to look to debug or fix this issue.  Could you provide some guidance?

Hi John-Paul,

It's not 100% clear to me at what dashboard you're looking at.

Provided you have toolkit installed there is a "perfSONAR Endpoint Pair Explorer" Grafana dashboard. On top of it there are drop-down list selection boxes where you can select Source and Destination hosts. At a time you can select one direction. In order to see opposite direction you need to swap hosts.

regards,

--

Szymon Trocha
Poznań Supercomputing & Networking Center

General NOC phone +48 61-858-2015 | noc.pcss.pl
Personal desk phone +48 61-858-2022

Wysłaliśmy do Ciebie ten e-mail w odpowiedzi na Twoje zapytanie lub w związku z oferowaną usługą. Przesłanie korespondencji do Centrum Zarządzania PCSS lub zgłoszenie telefoniczne jest równoznaczne z wyrażeniem zgody na przetwarzanie danych osobowych przez Instytut Chemii Bioorganicznej Polskiej Akademii Nauk w Poznaniu adres: ul. Z. Noskowskiego 12/14, 61-704 Poznań. Szczegółowe informacje znajdują się w naszej Polityce prywatności. | This message has been sent as a part of communication with PSNC NOC or your service request sent to us. For more information read our Privacy Policy.