perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Strelka, Justin writes:

 

I am setting up a new perfsonar host on 5.0.8 on RHEL 9 OS. I have set up some repeating tests on the box and have run into an issue where my tasks are all becoming Non-Starters after a period of time. I have two types of tasks running on the host currently latency and throughput. Below is an outline of both tasks submitted along with a screenshot of the pscheduler monitor command listing all/most new tasks as Non-Starters. Not sure what my issue is here.

 

Non-starters happen when one of the test participants (two of them for throughput, one for latency) is unable to find time on the schedule to run it or it’s being uncooperative for some other reason.  Looking at your monitor screenshot, it doesn’t look like there are schedule conflicts because there would be pending tasks among the non-starters.

 

You can diagnose this by looking at the schedule for a few minutes on either side of right now and picking out non-starting run (I set this one up to fail on purpose):

 

$ pscheduler schedule -PT5M +PT5M

…

2024-05-02T13:23:45Z - 2024-05-02T13:24:01Z (Non-Starter)

idleex --duration PT15S (Run with tool 'sleep')

https://work/pscheduler/tasks/069104b0-c936-40db-9a00-ea95ed2e0d08/runs/24c027be-f65f-46f6-9960-71b27f97c9a6

 

Then ask pScheduler what happened:

 

$ pscheduler result https://work/pscheduler/tasks/069104b0-c936-40db-9a00-ea95ed2e0d08/runs/24c027be-f65f-46f6-9960-71b27f97c9a6

2024-05-02T13:23:45+00:00 on work with sleep:

 

idleex --duration PT15S

 

Run never started: No times available for this run.

 

Since you have a lot of them, browsing the log (/var/log/pscheduler/pscheduler.log) will give you a bigger picture of why pScheduler is throwing in the towel on these runs:

 

May  2 13:23:46 work scheduler[430664]: scheduler INFO  48917: Posting non-starting run at 2024-05-02T13:23:45Z for task 069104b0-c936-40db-9a00-ea95ed2e0d08: No times available for this run.

 

Other things worth mentioning:

 

You’ll want to add the slip parameter to the schedule blocks in your pSConfig file.  The CLI uses a default slip of PT5M, going under the assumption that people running ad hoc measurements are willing to wait a few minutes for them to run.  pSConfig does not, which means that if it decides to schedule a measurement at 2:30, it happens then or not at all.   More slip decreases the chance of a collision with other runs, so add as much as you’re willing to tolerate.

 

Because running throughput at the same time as latency will distort the latency results, their plugins are configured to avoid each other.  There’s a variant of latency called latencybg that will run in parallel with anything else and produces a stream of results over a longer duration.  The reason we have the two types is a trade-off that lets people running the streaming flavor get results and can ignore distortions from overlapping throughput tests if they choose.  I bring this up because the latency tests you’re running will chew up about 55 seconds of schedule time each.  They’ll run in parallel with each other but are likely to collide with throughputs, especially if there’s no slip.

 

 

Hope that helps get you started.  Let us know what you find out from the logs on your system.

 

--Mark