perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Casey Russell writes:

 

     The 8 nodes in the mesh will just sporadically refuse to schedule some tests.  Right now it appears to be primarily throughput tests.  I end up with a bunch of "non-starting" tests in pscheduler, and logs like the ones below in pscheduler.log

 

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26599: Posting non-starting run at 2018-08-30T14:28:09Z for task 1a869753-f827-44bc-abb5-d0186075a482: ps-washburn-bw.perfsonar.kanren.net has no time available for this run

 

     As you can see, the misbehaving host is ps-wsu-bw.  It just suddenly begins to believe that most of the other hosts in the mesh have "no time available" for a test.  If I run a test manually, to one of the affected hosts, things seem to be fine (maybe it was a short term problem?).

 

When pScheduler gets a task that repeats, it’s going to start scheduling runs out to 24 hours, and as time passes it will schedule more.  If you’re seeing no-time errors, it means you have times when there are more tests to run than the system can find time to schedule without breaking any of the rules.  You will see this with throughput tests because they’re the only test that we schedule to have exclusive use of the system while they’re running.  Running a dozen traces at the same time isn’t an issue.  The log message says it was trying to schedule something tomorrow at 14:28, which tells me that’s where the congestion is.  If you run a test right now and there’s no congestion, it’ll run just fine.  There could stand to be more information in that message, so I’ve opened a ticket to expand on that:  https://github.com/perfsonar/pscheduler/issues/668.

 

In general, the best thing you can do for your tasks to make sure they get scheduled is add as much slip as you can tolerate.  I’d actually recommend against using random slip unless you have a measurement-related reason to use it.  With it turned on, runs will be scattered within the slip interval and could result in fragmentation that leaves gaps too small to squeeze in a measurement.  With it off, pScheduler will stack up tests one after the other as early as they can be scheduled and any available time within the slip interval is one big blob at the end.

 

There is no default slip for tasks submitted through the API; the CLI sets it to PT5M if none is explicitly provided.  Andy and I just had a short discussion about pSconfig, and he’ll follow up with his thoughts on that.  If you could forward us a copy of your mesh configuration off-list, we’ll have a look at it.

 

pScheduler has a little-known command called plot-schedule that can be used to produce a visualization of what the schedule looks like as a PNG.  Having just run it against your system, I suspect it may be buggy.  (There’s also a dependency-related problem on Debian systems where it doesn’t get a recent-enough version of Gnuplot.)  I’ll take a quick look at that and see if I can make it work correctly and will send you a plot of that host’s schedule at around the time where the congestion seems to be.  You can also use the schedule command (see “pscheduler schedule --help”) to look at the information textually.

 

HTH.

 

--Mark