perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Sorry, it might be helpful to see my maddash grid in case you'd like to see other failed tests in the grid.

http://ps-dashboard.kanren.net/maddash-webui

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Mon, Aug 28, 2017 at 2:17 PM, Casey Russell <> wrote:

Group,

     I've held off sending this to the group, because I was determined I was going to solve this one myself.  However, the beginning of the semester is upon us and I just haven't had the time to devote.  So here it goes, I'm asking for help.  This seems similar, but perhaps not the same as Mark Maciolek's current thread, but since I'm not certain, I didn't tie it to that thread.  

     I've got an entire Mesh that has started to randomly start failing tests.  What I mean by that is this.  Each day about 1/6 to 1/4 of the tests in the mesh will fail to run.  When a test between two hosts will fail, it will always fail at the same time of day... stay failed until exactly the same time the next day, then start working again.  And at that same time, a random sampling of other tests in the mesh will fail.  (because my hosts hate me apparently).  

     The first instance of failures I can find happened just after the 16th of August and my hosts are running auto updates.  Which is why I keyed in on Mark's post.  The failures start/swap each day just after noon.  When I use the API to look and see what failed with the run, I see either a very generic "participant-data-full" or (paraphrasing here) "participant data unavailable" timeout sort of error.

Here are some reference URLs

First instance I can find (just after the 16th of August)

http://ps-dashboard.kanren.net/maddash-webui/details.cgi?uri=/maddash/grids/KanREN-PS+-+IPv6+OWAMP+Latency/ps-fhsu-lt.perfsonar.kanren.net/ps-ku-lt.perfsonar.kanren.net/Loss

Current example: (failed at the moment 2:13pm CDT Aug 28th)

http://ps-dashboard.kanren.net/maddash-webui/details.cgi?uri=/maddash/grids/KanREN-PS+-+IPv6+OWAMP+Latency/ps-washburn-lt.perfsonar.kanren.net/ps-fhsu-lt.perfsonar.kanren.net/Loss

API "tasks" page:

https://ps-washburn-lt.perfsonar.kanren.net/pscheduler/tasks/35d52743-26e3-4432-b74e-fd899a6b5b45

API "runs" page:

https://ps-washburn-lt.perfsonar.kanren.net/pscheduler/tasks/35d52743-26e3-4432-b74e-fd899a6b5b45/runs

It seems like something is busy, or the API is temporarily unavailable, when the host (or hosts) are pulling the new Mesh and scheduling tests, but I've run dry trying to figure out how to troubleshoot the individual pieces of that.

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047