perfSONAR User Q&A and Other Discussion

[Pete Siemsen <>]

1.5 years later, and I have this problem again. When it arose a few days ago, I got frustrated and did a clean install. This time, I installed Debian 9.9, and installed perfsonar-toolkit 4.1.6 via apt-get. There are no tests configured and no maddash. Similar process counts to those I reported earlier in this thread:

perfsonar-1850$ ps -eTf | wc -l
2155
perfsonar-1850$ ps -eTf | grep pscheduler | wc -l
840
perfsonar-1850$ ps -eTf | grep powstream | wc -l
669
perfsonar-1850$ ps -eTf | grep owampd | wc -l
505
perfsonar-1850$ uptime

 11:56:23 up 1 day,  1:33,  2 users,  load average: 0.95, 1.11, 1.08

Seems crazy. This time around, the system is quite responsive, so I am simply going to raise the threshold from 2000 to 3000 and call it good.

-- Pete

On Tue, Dec 12, 2017 at 2:21 AM Marian Babik <> wrote:

Hi Mark,
coming back to this, the node is 4.0.2 on SL6. Last Friday I did the following:
[root@perfsonar-lt ~]# service pscheduler-runner stop
Starting pScheduler runner:                                [  OK  ]
[root@perfsonar-lt ~]# pkill -9 -f powstream
[root@perfsonar-lt ~]# service pscheduler-runner start
[root@perfsonar-lt ~]# ps -eTf | grep `id -u pscheduler` | wc -l
1644
[root@perfsonar-lt ~]# ps -eTf | grep `id -u pscheduler` | grep "powstream" | wc -l
379
[root@perfsonar-lt ~]# ps -eTf | grep `id -u pscheduler` | grep "runner" | wc -l
727

Today - so 4 days later I have:
[root@perfsonar-lt ~]# ps -eTf | grep `id -u pscheduler` | wc -l
2802
[root@perfsonar-lt ~]# ps -eTf | grep `id -u pscheduler` | grep "powstream" | wc -l
1113
[root@perfsonar-lt ~]# ps -eTf | grep `id -u pscheduler` | grep "runner" | wc -l
1033
[root@perfsonar-lt ~]# ps -eTf | grep `id -u pscheduler` | grep "\[run\]" | wc -l
619

The 619 [run] threads are all defunct, they show up like this:
501        774   774  9560  0 Nov29 ?        00:00:05 [run] <defunct>
etc.

I have also done reboots in the past and it didn’t help. It could be that this is SL6-only issue as Pete is also running SL6.

@Pete - user id of pscheduler could be different on your nodes, so grep 501 might not work, I have updated the commands to grep the correct id now

Cheers,
Marian

> On Dec 7, 2017, at 9:36 PM, Mark Feit <> wrote:
>
> Marian Babik writes:
>
>     …as stopping pscheduler runner leaves quite a number of powstream processes around…
>
> 4.0.2 includes some changes to the way the daemons and forked programs are started and stopped that should improve that situation.  Runners started by older versions won’t get swept up during upgrades because those versions don’t include the clean-ups, so you may have to go in and kill off the stragglers manually or just reboot.
>
> If you continue to see that on 4.0.2 systems after a flush-out of the old processes, drop me a line and we can have a look at it.
>
> --Mark
>
>