perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi Pete,

To echo some of what Ivan said, I don’t think a couple hundred powstreams should lead to you exceeding any thread ulimits, all of those processes are single threaded.  Do you have very many httpd processes?  I wonder if apache got stuck for some reason, it does some threading. What else is on the system? I assume esmond. Does it have maddash as well? That could also put some pressure on httpd. 

You could also try restarting apache (and might need to do a pkill if things are similarly stuck). Unfortunately you may also need to do a reboot because once you start exceeding ulimits it can break all sorts of stuff in unexpected ways. 

Thanks,

Andy

On December 6, 2017 at 4:27:06 AM, Marian Babik () wrote:

Hi Pete,
my understanding was that Andy suggested to restart pscheduler runner just to better understand where exactly to look for the root cause for the high number of threads/processes (as stopping pscheduler runner leaves quite a number of powstream processes around, it means there are indeed runaway processes that are no longer managed). I suspect it’s a bug that will need to be fixed, to be confirmed by Andy or Mark. As far as I can tell there is no workaround for it at the moment.

Just to check if this is perhaps OS specific, are you machines on centOS7 or still on SL6/or something else ?

Thanks,
Marian


> On Dec 5, 2017, at 8:03 PM, Pete Siemsen <> wrote:
>
> We monitor our perfSONAR machines with Nagios check-mk, which by default warns
> if there are more than 2000 threads on a machine. I'm getting that alarm. I
> found an email sequence where Andy suggested stopping/starting powstream. I
> did that but am still seeing "too many" threads. Should I raise the threshold
> above 2000?
>
> Here's what I did to no effect:
>
> perfsonar-1850# /etc/init.d/pscheduler-runner stop
> Stopping pScheduler runner: [ OK ]
> perfsonar-1850# /etc/init.d/pscheduler-runner status
> runner is stopped
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 357
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 354
> (waited 5 minutes)
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 354
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 353
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 352
> perfsonar-1850# pkill -9 -f powstream
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 9
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 9
> perfsonar-1850# /etc/init.d/pscheduler-runner start
> Starting pScheduler runner: [ OK ]
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 197
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 202
> perfsonar-1850# uptime
> 11:53:52 up 3 days, 21:50, 1 user, load average: 1.30, 1.02, 1.12
> (waited an hour)
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 203