perfSONAR User Q&A and Other Discussion

[Casey Russell <>]

Mark and Larry,

     I have seen this occasionally on my lower powered hosts, (and maybe on others, although I've been watching these lower powered hosts much closer, so I'm much more likely to have noticed it there.

     I don't have a host where that error is active, but here you can see where one of my hosts encountered the error yesterday (this was just before I installed the 4.0.2 beta on it and rebooted it). 

[root@ps-washburn-bw crussell]# cat /var/log/httpd/error_log | grep Max

[Mon Oct 16 16:50:42 2017] [error] server reached MaxClients setting, consider raising the MaxClients setting

[root@ps-washburn-bw crussell]# 

You can see that (today at least) these hosts do have a lot of connections open (sparing you the detailed output, although it's available if you want it).  Although that in an of its self is not necessarily a problem

[root@ps-washburn-bw crussell]# netstat -tan | wc -l

1238

(that's 1238 active TCP connections)

(Another of my lower powered hosts)

[crussell@ps-esu-bw ~]$ netstat -tan | wc -l

1903

Out of curiosity, I checked to see how many of those were hitting Apache on tcp port 80:

[root@ps-washburn-bw crussell]# netstat -an | grep ':80' | wc -l

78

[crussell@ps-esu-bw ~]$ netstat -tan | grep ':80' | wc -l

66

It doesn't seem too out of whack, but today may be entirely non-representative of what it looks like when the "MaxClients" problem was occurring.  Again, I installed the 4.0.2 beta on both of these hosts yesterday and the problem hasn't recurred since, so today's netstat results may not reflect what it looks like on an affected host.

Sincerely,

Casey Russell

Network Engineer

785-856-9809

2029 Becker Drive, Suite 282
Lawrence, Kansas 66047

On Tue, Oct 17, 2017 at 12:32 PM, Mark Feit <> wrote:

Larry Blunk writes:

 

Has anyone experienced Apache hitting the MaxClients limit and hanging?   We've had this happen on several boxes
since upgrading to 4.01.   We've had to restart Apache to get them functioning again.  We've upped the
MaxClients limit on them, but it still occurs even after doubling the setting to 512.   These are high perfomance
boxes, so it doesn't seem like it should be a CPU issue.

 

There shouldn’t be a lot of connections to the HTTP server during normal operations; MeshConfig and task setup from remote nodes are the only things that should be connecting.  The internal parts of pScheduler poke the database directly.

 

If you encounter that again, I’d be interested to see what the process table and netstat say about what’s connected and from where and if there are old processes that connect and aren’t dying.

 

--Mark