perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

My apologies Mark, I went ahead and rebooted that host about 11:00am CDT this morning to restore services to it.  So by the time you looked, it would have been fine.  

I've placed your script on that same host (and I'll probably follow up and put it on a couple of others).  We'll see if we can catch one in action.

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Fri, Oct 20, 2017 at 3:39 PM, Mark Feit <> wrote:

Casey Russell writes:

 

    One of my hosts (ps-ku-bw) has failed to schedule tasks today.  This is one of my larger hosts and the MaxClients problem might have actually been the trigger that began the avalanche.  I've left the host broken in case Mark or one of the other developers wants information from it while it's in this failed state.

 

Sorry it took me so long to get to this; I had some catching up to do after TechEx.

 

As of right now, it looks like that host is fine.  I watched it for several minutes using the monitor (pscheduler monitor --host ps-ku-bw.perfsonar.kanren.net) and saw lots of streaming latency running plus a steady diet of trace and the occasional throughput.

 

 At 9:47am yesterday, the httpd error log showed the following:

 

[root@ps-ku-bw crussell]# tail -f /var/log/httpd/error_log

…

[Wed Oct 18 09:47:29 2017] [error] server reached MaxClients setting, consider raising the MaxClients setting

 

Since pScheduler depends on Apache to do scheduling, I could see this being the cause of runs not being scheduled.  We should focus on what’s causing the number of connections to get that high.  I wrote up a short shell script that will ping pScheduler periodically and produce a netstat if the ping fails.  Leave it to run for awhile on the offending perfSONAR node and it should catch the conditions when Apache runs out of connections.  You can download it here:  https://gist.github.com/mfeit-internet2/24ad3bb83a6dd3fdef87fb9469f92a4a

 

As long as the central MA isn’t having this problem, any run that completes will be archived correctly since the local server isn’t involved in that process.

 

--Mark