perfSONAR User Q&A and Other Discussion

[Marian Babik <>]

Hi Ivan, Andy,
for me most of them account for pscheduler threads:
[root@perfsonar-lt ~]# ps -eTf | grep 501 | wc -l
1923
[root@perfsonar-lt ~]# id pscheduler
uid=501(pscheduler) gid=507(pscheduler) groups=507(pscheduler)

Most of which would be shared btw powstream and runner:
[root@perfsonar-lt ~]# ps -eTf | grep 501 | grep "powstream" | wc -l
711
[root@perfsonar-lt ~]# ps -eTf | grep 501 | grep "runner" | wc -l
711
Some in defunct [run]:
[root@perfsonar-lt ~]# ps -eTf | grep 501 | grep "\[run\]" | wc -l
468

— Marian


> On Dec 6, 2017, at 12:19 PM, Garnizov, Ivan (RRZE) 
> <>
>  wrote:
> 
> Hello Pete,
>  
> From your email it does not become clear how the system had reached more 
> than 2000 threads.
> The numbers, that you share tell us only about the significant number of 
> powstream processes. Still these are less than 1/5 of the 2000 threads.
> Do you have any suggestion about the rest of the processes?
> One possible problem might be coming from httpd daemon.
>  
> Please share the type of pS installation you have / a test-point or a 
> toolkit deployment / and check about the rest of the running processes.
>  
> Regards,
> Ivan Garnizov
>  
> GEANT SA1T2: pS deployments GN Operations
> GEANT SA2T3: pS development team
> GEANT SA3T5: eduPERT team
>  
>  
> -----Original Message-----
> From: 
> 
>  
> [mailto:]
>  On Behalf Of Marian Babik
> Sent: Mittwoch, 6. Dezember 2017 10:26
> To: Pete Siemsen
> Cc: 
> 
> Subject: Re: [perfsonar-user] more than 2000 threads
>  
> Hi Pete,
> my understanding was that Andy suggested to restart pscheduler runner just 
> to better understand where exactly to look for the root cause for the high 
> number of threads/processes (as stopping pscheduler runner leaves quite a 
> number of powstream processes around, it means there are indeed runaway 
> processes that are no longer managed). I suspect it’s a bug that will need 
> to be fixed, to be confirmed by Andy or Mark. As far as I can tell there is 
> no workaround for it at the moment.
>  
> Just to check if this is perhaps OS specific, are you machines on centOS7 
> or still on SL6/or something else ?
>  
> Thanks,
> Marian
>  
>  
> > On Dec 5, 2017, at 8:03 PM, Pete Siemsen 
> > <>
> >  wrote:
> >
> > We monitor our perfSONAR machines with Nagios check-mk, which by default 
> > warns
> > if there are more than 2000 threads on a machine. I'm getting that alarm. 
> > I
> > found an email sequence where Andy suggested stopping/starting powstream. 
> > I
> > did that but am still seeing "too many" threads. Should I raise the 
> > threshold
> > above 2000?
> >
> > Here's what I did to no effect:
> >
> > perfsonar-1850# /etc/init.d/pscheduler-runner stop
> > Stopping pScheduler runner:                                [  OK  ]
> > perfsonar-1850# /etc/init.d/pscheduler-runner status
> > runner is stopped
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 357
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 354
> > (waited 5 minutes)
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 354
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 353
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 352
> > perfsonar-1850# pkill -9 -f powstream
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 9
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 9
> > perfsonar-1850# /etc/init.d/pscheduler-runner start
> > Starting pScheduler runner:                                [  OK  ]
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 197
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 202
> > perfsonar-1850# uptime
> > 11:53:52 up 3 days, 21:50,  1 user,  load average: 1.30, 1.02, 1.12
> > (waited an hour)
> > perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l
> > 203

Attachment: smime.p7s
Description: S/MIME cryptographic signature