perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Group,

     Over the summer, we upgraded hardware on all 8 of our nodes (CPU and memory), installed them fresh with CentOS 7 and PS 4.0 and rebuilt our mesh with the new PSconfig tools a few weeks ago when 4.1 came out.  

     For a few glorious weeks (when all the nodes were upgraded, but before the 4.1 upgrades) I had a green dashboard and thought all was well with the world.  I can't say for sure it was the introduction of 4.1, but something in the last 2 weeks has put me right back where I was before when I thought my primary problem was underpowered nodes.  

     The 8 nodes in the mesh will just sporadically refuse to schedule some tests.  Right now it appears to be primarily throughput tests.  I end up with a bunch of "non-starting" tests in pscheduler, and logs like the ones below in pscheduler.log

Aug 29 09:28:09 ps-wsu-bw journal: runner INFO     10012256: Running https://ps-wsu-bw.perfsonar.kanren.net/pscheduler/tasks/1e3e12f4-2d58-4097-86e2-dc0b014cb964/runs/6425a5f3-df6f-4e5a-870c-5dbdbd835f3d

Aug 29 09:28:09 ps-wsu-bw journal: runner INFO     10012256: With iperf3: throughput --bandwidth 920000000 --duration PT10S --source ps-wsu-bw.perfsonar.kanren.net --ip-version 4 --dest ps-ku-bw.perfsonar.kanren.net --source-node ps-wsu-bw.perfsonar.kanren.net --dest-node ps-ku-bw.perfsonar.kanren.net --udp

Aug 29 09:28:11 ps-wsu-bw journal: runner WARNING  10012256: Starting 0:00:02.632591 later than scheduled

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26599: Posting non-starting run at 2018-08-30T14:28:09Z for task 1a869753-f827-44bc-abb5-d0186075a482: ps-washburn-bw.perfsonar.kanren.net has no time available for this run

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26644: Posting non-starting run at 2018-08-30T14:28:09Z for task f6954127-5cab-4279-a61a-269c095e7426: ps-esu-bw.perfsonar.kanren.net has no time available for this run

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26643: Posting non-starting run at 2018-08-30T14:28:09Z for task 06fb5795-bbe8-4d5e-8c5b-7696e42637db: ps-ku-bw.perfsonar.kanren.net has no time available for this run

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26595: Posting non-starting run at 2018-08-30T14:28:09Z for task 31a885b9-54c5-46ca-b1ec-c1935e13058e: ps-ksu-bw.perfsonar.kanren.net has no time available for this run

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26596: Posting non-starting run at 2018-08-30T14:28:09Z for task 61482a12-3ecc-4a68-a241-49906390f7b7: ps-ku-bw.perfsonar.kanren.net has no time available for this run

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26645: Posting non-starting run at 2018-08-30T14:28:09Z for task 047cb255-b64a-4be5-89b6-2b4a1062a924: ps-psu-bw.perfsonar.kanren.net has no time available for this run

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26598: Posting non-starting run at 2018-08-30T14:28:09Z for task 93f8a914-9b8a-4d93-8088-12b5b0f2b647: ps-psu-bw.perfsonar.kanren.net has no time available for this run

Aug 29 09:28:37 ps-wsu-bw journal: scheduler INFO     26642: Posting non-starting run at 2018-08-30T14:28:09Z for task 611ab2f6-6b46-47ce-9e9c-f6c2e00c1387: ps-ksu-bw.perfsonar.kanren.net has no time available for this run

     As you can see, the misbehaving host is ps-wsu-bw.  It just suddenly begins to believe that most of the other hosts in the mesh have "no time available" for a test.  If I run a test manually, to one of the affected hosts, things seem to be fine (maybe it was a short term problem?).

     The web interface no longer tells me what percentage of the time that throughput tests will be running, but my mesh config ( I think) seems sane for these hosts.  Looking at a bandwidth graph (10s resolution) shows lots of dead time for the bandwidth interfaces on these boxes.  

     I suppose it could be that for just a very short duration, there is no time available.  Especially if the hosts get synced up and are all pulling their mesh configs and trying to schedule their tests at roughly the same time.  I just went in today and added a slip (and sliprand) to all of my schedules in the mesh to see if that helps.  Does anyone have any idea what else I should look for?  have you seen this before?

I'm happy to share any other info or logs if you want them.  

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047