perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan (RRZE)" <>]

Hi Phil,

 

It appears to me, you have some MaDDash configuration issue, which results in increased Apache load, which accumulates with time and in the end crashes Apache.

Please start monitoring the usage of memory and cpu on the host and by Apache in attempt to relate to the problem to the issue.

 

The numerous Nagios check failures result in numerous Apache requests and if these timeout, then they put load on the Apache.

If there are Nagios check failures, these should be observed on the GUI as well. Is this true?

 

The simplest example: You have left the “example” MaDDash config lines in the maddash.yaml file.

Please check all the dashboards you may have and seek for “no data” cells.

 

WRT events in “/var/log/httpd/error_log”

Unfortunately the normal operation of Cassandra and Esmond lead to these false positives in the Apache log. Please note these are regular on the hour.

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

Jubiläumsjahr 2018 - IT in Bewegung

Das RRZE - der IT-Dienstleister der FAU

www.50-jahre.rrze.fau.de

 

Von: Phil Reese [mailto:]
Gesendet: Dienstag, 7. August 2018 18:10
An: Garnizov, Ivan (RRZE) <>; Andrew Lake <>;
Betreff: Re: AW: AW: [perfsonar-user] Maddash web server woes

 

Hi Ivan,

Thanks for being willing to work with me on this.

Below are some log outputs which might be of help.

Phil


First this MA server is dedicated to just this single service.  There are 13 PS testpoint systems feeding data to the server in a full mesh config.  There is a lot of DNS traffic so I think it is keeping the server pretty busy but it seems like it should work and not kill the httpd (apache) process.

I made these updates to the Apache Config, without much impact:
<IfModule prefork.c>
    StartServers        25
    MinSpareServers     40
    MaxSpareServers     70
    MaxRequestWorkers   640
    MaxConnectionsPerChild 7500
</IfModule>

From /var/log/httpd/error_log:

Various assortment of these:
[Tue Aug 07 08:25:37.943206 2018] [mpm_worker:notice] [pid 8873:tid 139689130432640] AH00296: caught SIGWINCH, shutting down gracefully
[Tue Aug 07 08:27:08.334259 2018] [suexec:notice] [pid 28354:tid 140127999797376] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Aug 07 08:27:08.335219 2018] [ssl:warn] [pid 28354:tid 140127999797376] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Aug 07 08:27:08.353946 2018] [auth_digest:notice] [pid 28354:tid 140127999797376] AH01757: generating secret for digest authentication ...
[Tue Aug 07 08:27:08.354430 2018] [lbmethod_heartbeat:notice] [pid 28354:tid 140127999797376] AH02282: No slotmem from mod_heartmonitor

Lots of these:
[Tue Aug 07 08:27:09.246540 2018] [:error] [pid 28358:tid 140127691790080] path= ['/usr/lib/esmond/src/dlnetsnmp/lib', '/usr/lib/esmond', '/usr/lib/esmond/esmond_client', '/usr/lib/esmond/esmond', '/usr/lib/esmond/lib/python2.7', '/usr/lib/esmond/lib/python2.7/site-packages', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages']
[Tue Aug 07 08:27:09.294315 2018] [:error] [pid 28359:tid 140127691790080] path= ['/usr/lib/esmond/src/dlnetsnmp/lib', '/usr/lib/esmond', '/usr/lib/esmond/esmond_client', '/usr/lib/esmond/esmond', '/usr/lib/esmond/lib/python2.7', '/usr/lib/esmond/lib/python2.7/site-packages', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages']

These started after a httpd restart:
[Tue Aug 07 08:27:11.967617 2018] [:error] [pid 28363:tid 140127683397376] cassandra_db [INFO] Checking/creating column families
[Tue Aug 07 08:27:11.968883 2018] [:error] [pid 28363:tid 140127683397376] cassandra_db [INFO] Schema check done
(lots more)

---
maddash-server.log details:
These come in and out:
INFO 2018-08-01 00:00:00,001 oldestAllowedTime=1532502000
ERROR 2018-08-01 09:38:25,385 Error scheduling job The Scheduler has been shutdown.
ERROR 2018-08-01 09:38:25,416 Error executing CheckSchedulerJob: The Scheduler has been shutdown.
INFO 2018-08-01 12:00:00,002 oldestAllowedTime=1532545200
INFO 2018-08-02 00:00:00,001 oldestAllowedTime=1532588400

These seem like the INFO lines suggest good stuff, but the Nagios checks are concerning:
INFO 2018-08-04 12:00:00,001 oldestAllowedTime=1532804400
INFO 2018-08-05 00:00:00,002 oldestAllowedTime=1532847600
ERROR 2018-08-05 11:51:47,803 Error running nagios check: null
ERROR 2018-08-05 11:55:44,599 Error running nagios check: null
ERROR 2018-08-05 11:55:44,603 Error running nagios check: null
ERROR 2018-08-05 11:55:44,604 Error running nagios check: null
ERROR 2018-08-05 11:55:44,617 Error running nagios check: null
ERROR 2018-08-05 11:55:44,619 Error running nagios check: null
ERROR 2018-08-05 11:55:44,620 Error running nagios check: null
ERROR 2018-08-05 11:55:44,621 Error running nagios check: null
ERROR 2018-08-05 11:55:44,622 Error running nagios check: null
ERROR 2018-08-05 11:55:44,623 Error running nagios check: null
ERROR 2018-08-05 11:55:44,624 Error running nagios check: null
ERROR 2018-08-05 11:55:44,624 Error running nagios check: null
ERROR 2018-08-05 11:55:44,624 Error running nagios check: null
ERROR 2018-08-05 11:55:44,624 Error running nagios check: null
ERROR 2018-08-05 11:55:44,627 Error running nagios check: null
ERROR 2018-08-05 11:55:44,628 Error running nagios check: null
ERROR 2018-08-05 11:55:44,629 Error running nagios check: null
ERROR 2018-08-05 11:55:44,602 Error running nagios check: null
ERROR 2018-08-05 11:55:44,602 Error running nagios check: null
ERROR 2018-08-05 11:55:44,632 Error running nagios check: null
INFO 2018-08-05 12:00:00,002 oldestAllowedTime=1532890800
INFO 2018-08-06 00:00:00,002 oldestAllowedTime=1532934000

On 8/7/18 2:19 AM, Garnizov, Ivan (RRZE) wrote:

Hi Phil,

 

Actually it is the normal behavior of CentOS 7.5 to have the Apache daemon run with this flag –DFOREGROUND.

We need to understand, where the problem stems from.

If Appache is opening so many connections, then probably your sever is receiving many requests (hopefully not too many)

Please try to restrict the access in order to see if that will make the system stable again.

 

Also in the logs of Apache there might be references to issues, which are related to the problem.

Please send logs of Apache from the server to see, if there are issues related to MaDDash operation.

 

Best regards,

Ivan