perfSONAR User Q&A and Other Discussion

["Pennington, Mike" <>]

Thanks Ivan (and thanks Mark, about to respond to your email as well).  There is no config management automation stuff on here, it’s pretty basic.

 

I’ve attached the logs zipped up, they are pretty huge.

 

 

Mike Pennington

Network Engineer

Connecticut Education Network (CEN)

860-622-4566

www.cen.ct.gov

 

From: Ivan Garnizov [mailto:]
Sent: Wednesday, December 13, 2017 5:29 AM
To: ; Pennington, Mike <>;
Subject: Re: [perfsonar-user] Best way to troubleshoot HTTP/HTTPS flapping?

 

Hello Mike,

In order for us to more efficiently assist you please send us the reported events in the /var/log/httpd/ folder files: ssl_error_log and ssl_access_log

Please also share, if there is any kind of configuration management automation operational on the server.

Regards,
Ivan Garnizov

On 12 December 2017 21:16:24 CET, "Pennington, Mike" <> wrote:

One of our Perfsonar nodes has been acting funky over the past week or so.  Our monitoring shows HTTP/HTTPS going up and down at random times during the day, I’ve confirmed it is actually going down as I cannot access perfsonar-hartford.cen.ct.gov during those times.  It can be anywhere from a few minutes and has lasted up to 30 minutes.  I’ve rebooted multiple times, power cycled it, yum is all up to date, etc.  Tests and results seem fine, it just seems to be HTTP/HTTPS access.  I’ve looked through some of the httpd logs but nothing is jumping out at me. 

 

I can still SSH into the box no problem during the times HTTP/HTTPS is down.

 

Seems to be plenty of space left on the disk, nothing crazy when running TOP either:

 

top - 15:14:36 up  2:32,  1 user,  load average: 2.17, 2.25, 3.03

Tasks: 1238 total,   1 running, 1199 sleeping,   0 stopped,  38 zombie

Cpu(s):  6.3%us,  7.0%sy,  0.0%ni, 63.7%id, 22.7%wa,  0.0%hi,  0.3%si,  0.0%st

Mem:   8007548k total,  7837420k used,   170128k free,    11588k buffers

Swap:  8142844k total,   900176k used,  7242668k free,  2535348k cached

 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

2779 postgres  20   0  525m 165m 142m D  9.2  2.1   3:07.64 postmaster

2772 postgres  20   0  339m 139m 136m S  8.8  1.8   5:30.88 postmaster

2767 pschedul  20   0 2749m  11m 1708 S  3.6  0.1  20:31.29 archiver

2773 postgres  20   0  339m 111m 107m S  3.6  1.4   3:05.44 postmaster

25995 root      20   0 14064 2084  892 R  1.6  0.0   0:00.23 top

   80 root      20   0     0    0    0 S  1.3  0.0   1:06.28 kswapd0

2770 pschedul  20   0 3861m  42m 2616 S  1.3  0.5   2:36.86 runner

2771 postgres  20   0  338m 143m 140m S  1.3  1.8   8:33.51 postmaster

25522 postgres  20   0  343m  26m  21m S  1.0  0.3   0:00.49 postmaster

1628 cassandr  20   0 5628m 1.7g 4476 S  0.7 22.7   6:42.32 java

26092 postgres  20   0  343m  18m  13m S  0.7  0.2   0:00.10 postmaster

1812 memcache  20   0  322m  14m  372 S  0.3  0.2   0:15.38 memcached

1841 npad      20   0  322m 1956 1008 S  0.3  0.0   0:02.50 DiagServer.py

2765 pschedul  20   0 2278m  25m 2596 S  0.3  0.3   0:25.61 scheduler

 

When I go to restart httpd, it actually has to stop it so I know the process is actually running:

 

[root@ps-10G-dual-core-hartford init.d]# ./httpd restart

Stopping httpd:                                            [  OK  ]

Starting httpd:                                            [  OK  ]

[root@ps-10G-dual-core-hartford init.d]#

 

 

Has anyone else had this problem or know the best way to start troubleshooting? 

 

Thanks!

 

Mike Pennington

Network Engineer

Connecticut Education Network (CEN)

860-622-4566

www.cen.ct.gov

 

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Attachment: logs.zip
Description: logs.zip