perfSONAR User Q&A and Other Discussion

[Marian Babik <>]

Hi Pete,
my understanding was that Andy suggested to restart pscheduler runner just to 
better understand where exactly to look for the root cause for the high 
number of threads/processes (as stopping pscheduler runner leaves quite a 
number of powstream processes around, it means there are indeed runaway 
processes that are no longer managed). I suspect it’s a bug that will need to 
be fixed, to be confirmed by Andy or Mark. As far as I can tell there is no 
workaround for it at the moment. 

Just to check if this is perhaps OS specific, are you machines on centOS7 or 
still on SL6/or something else ? 

Thanks,
Marian


> On Dec 5, 2017, at 8:03 PM, Pete Siemsen 
> <>
>  wrote:
> 
> We monitor our perfSONAR machines with Nagios check-mk, which by default 
> warns
> if there are more than 2000 threads on a machine. I'm getting that alarm. I
> found an email sequence where Andy suggested stopping/starting powstream. I
> did that but am still seeing "too many" threads. Should I raise the 
> threshold
> above 2000?
> 
> Here's what I did to no effect:
> 
> perfsonar-1850# /etc/init.d/pscheduler-runner stop
> Stopping pScheduler runner:                                [  OK  ]
> perfsonar-1850# /etc/init.d/pscheduler-runner status
> runner is stopped
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 357
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 354
> (waited 5 minutes)
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 354
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 353
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 352
> perfsonar-1850# pkill -9 -f powstream
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 9
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 9
> perfsonar-1850# /etc/init.d/pscheduler-runner start
> Starting pScheduler runner:                                [  OK  ]
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 197
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 202
> perfsonar-1850# uptime
> 11:53:52 up 3 days, 21:50,  1 user,  load average: 1.30, 1.02, 1.12
> (waited an hour)
> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> 203

Attachment: smime.p7s
Description: S/MIME cryptographic signature