perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan (RRZE)" <>]

Hello Pete,

 

From your email it does not become clear how the system had reached more than 2000 threads.

The numbers, that you share tell us only about the significant number of powstream processes. Still these are less than 1/5 of the 2000 threads.

Do you have any suggestion about the rest of the processes?

One possible problem might be coming from httpd daemon.

 

Please share the type of pS installation you have / a test-point or a toolkit deployment / and check about the rest of the running processes.

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

 

-----Original Message-----
From: [] On Behalf Of Marian Babik
Sent: Mittwoch, 6. Dezember 2017 10:26
To: Pete Siemsen
Cc:
Subject: Re: [perfsonar-user] more than 2000 threads

 

Hi Pete,

my understanding was that Andy suggested to restart pscheduler runner just to better understand where exactly to look for the root cause for the high number of threads/processes (as stopping pscheduler runner leaves quite a number of powstream processes around, it means there are indeed runaway processes that are no longer managed). I suspect it’s a bug that will need to be fixed, to be confirmed by Andy or Mark. As far as I can tell there is no workaround for it at the moment.

 

Just to check if this is perhaps OS specific, are you machines on centOS7 or still on SL6/or something else ?

 

Thanks,

Marian

 

 

> On Dec 5, 2017, at 8:03 PM, Pete Siemsen <> wrote:

>

> We monitor our perfSONAR machines with Nagios check-mk, which by default warns

> if there are more than 2000 threads on a machine. I'm getting that alarm. I

> found an email sequence where Andy suggested stopping/starting powstream. I

> did that but am still seeing "too many" threads. Should I raise the threshold

> above 2000?

>

> Here's what I did to no effect:

>

> perfsonar-1850# /etc/init.d/pscheduler-runner stop

> Stopping pScheduler runner:                                [  OK  ]

> perfsonar-1850# /etc/init.d/pscheduler-runner status

> runner is stopped

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 357

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 354

> (waited 5 minutes)

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 354

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 353

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 352

> perfsonar-1850# pkill -9 -f powstream

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 9

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 9

> perfsonar-1850# /etc/init.d/pscheduler-runner start

> Starting pScheduler runner:                                [  OK  ]

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 197

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 202

> perfsonar-1850# uptime

> 11:53:52 up 3 days, 21:50,  1 user,  load average: 1.30, 1.02, 1.12

> (waited an hour)

> perfsonar-1850# ps -ef | grep "/usr/bin/powstream" | wc -l

> 203