perfSONAR User Q&A and Other Discussion

["Robinson, John-Paul" <>]

Is there a way to turn on the debug logging in /usr/lib/pscheduler/classes/tool/iperf3/run via a config setting?

---

From: Robinson, John-Paul <>
Sent: Sunday, August 11, 2024 12:02 PM
To: <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

I think I've narrowed down the issue further.  I'm now limiting my configuration to just trying to do a bidirectional test from one of the nodes to the other.  I'm directly workding with the psconfig template created by the web ui.

What appears to be happening is that psconfig correctly defines the bidirectional tests but when they get registered with the lead host they are end up both being  defined in the same direction.

Here's the transaction log that shows the bidirectional tests getting created.  Based on my understanding of the docs,  the lead_url is where the test should be registered.

2024-08-11 10:48:46 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" msg=Creating tasks

2024-08-11 10:48:47 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" checksum=rLl0+VaPhR0XGD63IMRS+g lead_url=https://192.168.1.53/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.93", "dest-node": "192.168.1.93", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.53", "source-node": "192.168.1.53", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-12T15:48:46Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-11 10:48:55 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" checksum=9n9gQQ4gmO+yddcUw6EyOA lead_url=https://192.168.1.93/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.53", "dest-node": "192.168.1.53", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.93", "source-node": "192.168.1.93", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-12T15:48:46Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-11 10:49:02 INFO guid=35f210b8-1345-4b0c-9885-d59f4dd6466f action="create" msg=Done creating tasks

When I look at the tests that are scheduled, however, both hosts have a source and destination that are the same, leading to only a unidirectional test.

Here's the schedule from the ps-521-int machine which is configured with the bidirectional test template

ubuntu@ps-521-int:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-11T16:45:25+00:00 - 2024-08-11T16:45:54+00:00 (Scheduling)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521-int/pscheduler/tasks/9e3b8ea9-49f5-4e66-8aee-659e8d0aedee/runs/7c70dd83-4bc7-40af-8c56-a3bac6c231ba

Here's the schedule from the peer node ps-521, note the source and dest are the same as above.

ubuntu@ps-521:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-11T16:45:25+00:00 - 2024-08-11T16:45:54+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521/pscheduler/tasks/9e3b8ea9-49f5-4e66-8aee-659e8d0aedee/runs/7c70dd83-4bc7-40af-8c56-a3bac6c231ba

I'm guessing these are the same tests, but that makes me wonder why i'm not seeing a similar pair for the second task in the agent transaction log above.

Thanks for any guidance.

PS. I found the debug logging feature for the pscheduler-api. 🙂

touch /var/pscheduler-server/pscheduler-api-debug

journalctl -u apache2

I'll let you know if I find anything interesting.

---

From: <> on behalf of "Robinson, John-Paul" <>
Sent: Saturday, August 10, 2024 1:32 PM
To: <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

Hi Folks,

I've done some further investigation.  I cleared out my back log of failed tests and then set up just one of the two nodes to do the bi-directional tests.

On the node 192.168.1.53 on which I re-enabled the bidirectional tests, I see the following in the/var/log/perfsonar/psconfig-pscheduler-agent-transactions.log

2024-08-10 12:59:57 INFO guid=a07f9d62-64cf-46ad-b3e4-389d0d968bcd action="create" checksum=rLl0+VaPhR0XGD63IMRS+g lead_url=https://192.168.1.53/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.93", "dest-node": "192.168.1.93", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.53", "source-node": "192.168.1.53", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-11T17:59:57Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-10 13:00:05 INFO guid=a07f9d62-64cf-46ad-b3e4-389d0d968bcd action="create" checksum=9n9gQQ4gmO+yddcUw6EyOA lead_url=https://192.168.1.93/pscheduler test_type=throughput task={"test": {"spec": {"dest": "192.168.1.53", "dest-node": "192.168.1.53", "duration": "PT20S", "ip-version": 4, "source": "192.168.1.93", "source-node": "192.168.1.93", "schema": 1}, "type": "throughput"}, "archives": [{"archiver": "http", "data": {"schema": 2, "_url": "http://localhost/logstash", "op": "put", "_headers": {"x-ps-observer": "192.168.1.53", "content-type": "application/json", "Authorization": "Basic cGVyZnNvbmFyOktVYWM4MEZMRG5KYnNnRFh6QXdZ"}}}], "schedule": {"repeat": "PT3600S", "slip": "PT3600S", "sliprand": true, "until": "2024-08-11T17:59:57Z"}, "tools": ["bwctliperf3", "iperf3"], "reference": {"psconfig": {"created-by": {"uuid": "96A99843-8A1F-40E1-BDD7-FAE8A148A247", "user-agent": "psconfig-pscheduler-agent", "agent-hostname": "ps-521-int"}}}}

2024-08-10 13:00:13 INFO guid=a07f9d62-64cf-46ad-b3e4-389d0d968bcd action="create" msg=Done creating tasks

This looks like the bi-directional tests get set up.  They should run a throughput once per hour.

When I look at the test schedule on both of the nodes I don't see any bi-directional tests scheduled.  They are all just one directional tests.

Here are the tests on the node on which the tests were configured (192.169.1.53)

ubuntu@ps-521-int:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-10T18:56:53+00:00 - 2024-08-10T18:57:22+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521-int/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/12ee00ea-23a4-42a5-81dd-38b105a0ef12

2024-08-10T19:59:24+00:00 - 2024-08-10T19:59:53+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521-int/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/4fd580bd-c84d-43c4-9331-f7e5a2ad0dfe

Here are the tests on the node which is the target of the tests (192.168.1.93)

ubuntu@ps-521:~$ pscheduler schedule --filter-test=throughput +PT2H

2024-08-10T18:56:53+00:00 - 2024-08-10T18:57:22+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/12ee00ea-23a4-42a5-81dd-38b105a0ef12

2024-08-10T19:59:24+00:00 - 2024-08-10T19:59:53+00:00 (Pending)

throughput --source 192.168.1.53 --source-node 192.168.1.53 --dest 192.168.1.93 --dest-node 192.168.1.93 --duration PT20S --ip-

  version 4 (Run with tool 'iperf3')

https://ps-521/pscheduler/tasks/e6402d67-65b8-4866-a2f7-9c575783d6fb/runs/4fd580bd-c84d-43c4-9331-f7e5a2ad0dfe

As you can see, all the tests are outbound from 192.168.1.53.  I can give a bigger time window but it doesn't show any reverse tests.

I'm not sure where to look next.  Is there another log related to /var/log/perfsonar/psconfig-pscheduler-agent-transactions.log that might be helpful in understanding why the reverse tests appears to be schedule but then doesn't make it into the schedule?

Thanks,

John-Paul

---

From: Robinson, John-Paul <>
Sent: Wednesday, August 7, 2024 12:43 PM
To: <>
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

 

Hi,

Thanks for the feedback.  I'm working with perfsonar 5.1.2 so it's the grafana interface.   I navigated to the the pair explorer plot you mentioned.  I see all my hosts I have tests set up in the destination drop-down, but I only see my own hosts in the source.  So I can't swap the tests results around. 

---

From: Szymon Trocha
Sent: Wednesday, August 7, 2024 6:00 AM
To: Robinson, John-Paul
Cc: 
Subject: Re: [perfsonar-user] Bi directional test aren't plotting

W dniu 05.08.2024 o 01:41, "Robinson, John-Paul" (via perfsonar-user Mailing List) pisze:

Hi,

I'm wondering what steps are necessary to enable the plotting of the forward and reverse thoughput results in the web UI.   This may actually be rooted in an issue of those results not being logged or the tests not being performed. In any case, I'd like to fix the problem.

In ps releases prior to 5.x I was able to just set up tests and select "send and receive" in the test set up UI and the results would show up for both directions of successfully completed tests.  In all my setups since switching to 5x, this has not worked out of the box.

I did a fresh install of 5.1.2 on two test boxes on Ubuntu 22.04 on the same lan segment.  I set up a throughput test on each one pointing at the other node.  I selected the "Send and Recieve" in the advance testing parameters.   When I look at the plots I only see the tests that are outbound from the host.  I never see the reverse (inbound) test results. I can run tests in both directions from the command line using pscheduler.

I'm not sure where to look to debug or fix this issue.  Could you provide some guidance?

Hi John-Paul,

It's not 100% clear to me at what dashboard you're looking at.

Provided you have toolkit installed there is a "perfSONAR Endpoint Pair Explorer" Grafana dashboard. On top of it there are drop-down list selection boxes where you can select Source and Destination hosts. At a time you can select one direction. In order to see opposite direction you need to swap hosts.

regards,

--

Szymon Trocha
Poznań Supercomputing & Networking Center

General NOC phone +48 61-858-2015 | noc.pcss.pl
Personal desk phone +48 61-858-2022

Wysłaliśmy do Ciebie ten e-mail w odpowiedzi na Twoje zapytanie lub w związku z oferowaną usługą. Przesłanie korespondencji do Centrum Zarządzania PCSS lub zgłoszenie telefoniczne jest równoznaczne z wyrażeniem zgody na przetwarzanie danych osobowych przez Instytut Chemii Bioorganicznej Polskiej Akademii Nauk w Poznaniu adres: ul. Z. Noskowskiego 12/14, 61-704 Poznań. Szczegółowe informacje znajdują się w naszej Polityce prywatności. | This message has been sent as a part of communication with PSNC NOC or your service request sent to us. For more information read our Privacy Policy.