perfSONAR User Q&A and Other Discussion

[Mark Feit <>]

Victor Olifer writes:
    
    It happened the second time over couple of weeks that  pscheduler on our
    production pS host all of a sudden stopped working properly and reporting
    error messages like this:
    
    May 13 11:08:58 ps-londhx journal: pscheduler-api ERROR    Internal error
    2001:630:3c:f800::209 POST https://ps-londhx1.ja.net/pscheduler/tasks: 
Error
    getting tools from ps-londhx1.ja.net: Operation timed out after 10001
    milliseconds with 0 out of -1 bytes received

I had a look at your system as much as I can from the outside.  pScheduler 
doesn't appear to have an excessive workload, but the API is taking an 
excessive (although inconsistent) amount of time to answer queries:

$ time curl -sk https://ps-londhx1.ja.net/pscheduler
"This is the pScheduler API server on ps-londhx1.ja.net (ps-londhx1.ja.net)."
real    0m9.563s

This is the simplest query the API can answer, requiring only a round trip 
through the web server.  The toolkit pages respond quickly, which means that 
the system as a whole is probably healthy.  There are limited processes and 
threads available in the web server for pScheduler requests; running beyond 
that limit would cause requests to back up.  Is there anything in the web 
server logs that suggests any of the endpoints under /pscheduler are seeing 
large amounts of activity?

For anything beyond that, drop me a line off-list and we can get together on 
video to give it a look.

--Mark