Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] Lots of "worker-still-working" log messages

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] Lots of "worker-still-working" log messages


Chronological Thread 
  • From: Aaron Brown <>
  • To: Alan Whinery <>
  • Cc: "" <>
  • Subject: Re: [perfsonar-user] Lots of "worker-still-working" log messages
  • Date: Wed, 4 Mar 2015 18:59:51 +0000
  • Accept-language: en-US
  • Authentication-results: hawaii.edu; dkim=none (message not signed) header.d=none;
Do a “find” on that directory. The IPC::DirQueue module creates a bunch of
deeply nested directories/files.

Cheers,
Aaron

> On Mar 4, 2015, at 1:44 PM, Alan Whinery
> <>
> wrote:
>
> On 3/4/2015 8:40 AM, Alan Whinery wrote:
>> On 3/4/2015 8:28 AM, Aaron Brown wrote:
>>> Hey Alan,
>>>
>>> Are there a bunch of files in /var/lib/perfsonar/regular_testing?
>>>
>>> Cheers,
>>> Aaron
> Or perhaps I should have said directories...
>
> "ls *" gets 125 directories, 54 regular files.
>
> [root@uhmanoa-dl
> regular_testing]# ls -l
> total 500
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:11 bwctl_0BQRw
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:17 bwctl_0QH8E
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:32 bwctl_5k0Wq
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:36 bwctl_71FtU
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 bwctl_AhVhc
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:07 bwctl_E9aEh
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:01 bwctl_EV9Z6
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:07 bwctl_EYlKB
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 08:58 bwctl_FlgvC
> drwx------ 2 perfsonar perfsonar 4096 Feb 19 18:47 bwctl_gBppB
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:15 bwctl_kMDVi
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:38 bwctl_l4ksQ
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:07 bwctl_Mybgb
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:38 bwctl_QzaBA
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:34 bwctl_SAF_T
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:31 bwctl_TljeT
> drwx------ 2 perfsonar perfsonar 4096 Feb 19 18:45 bwctl_u3Mum
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:36 bwctl_xOkeh
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:33 bwctl_XSDvn
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:16 bwctl_z4m5O
> drwxr-x--- 4 perfsonar perfsonar 4096 Dec 9 13:11 esmond_latency_localhost
> drwxr-x--- 4 perfsonar perfsonar 4096 Dec 9 13:11
> esmond_throughput_localhost
> drwxr-x--- 4 perfsonar perfsonar 4096 Dec 9 13:11
> esmond_traceroute_localhost
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_1723R
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_2hu2R
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_3eKCy
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_3zqqB
> drwx------ 2 perfsonar perfsonar 4096 Dec 24 13:59 owamp_46rTZ
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_5JObr
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_5x2ua
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_65dDj
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_6Jvkt
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_6TEyC
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_7jw_T
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_7qHIi
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9hu1N
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9nYtA
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_9rE5X
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9sD7v
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9TSfJ
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_AKoMY
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:07 owamp_AYzch
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_AZRoN
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_bBAWm
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:07 owamp_BPHTO
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_bSKCn
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_cH9GH
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_ckzKy
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_CpoHI
> drwx------ 2 perfsonar perfsonar 4096 Jan 13 02:00 owamp_DH3y4
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_D_nzz
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_E0uGT
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_e5Qff
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_f0BZ7
> drwx------ 2 perfsonar perfsonar 4096 Jan 13 02:00 owamp_f8e2s
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_FEodt
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fFGwc
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fjTEN
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:07 owamp_fTuzd
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fuu1J
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_fUwhB
> drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:47 owamp_Fvurb
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fw0nn
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_GI0vV
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_GlieD
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_GO2E6
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_guJHF
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_hmofi
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_iaPC0
> drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:41 owamp_j7HFV
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_jaidn
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_k680N
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_kHXyR
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_kSyX4
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_Ky6te
> drwx------ 2 perfsonar perfsonar 4096 Dec 24 14:00 owamp_L3zmI
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_lBYgk
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_LkSyt
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_lLC2x
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_Lt422
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_lW5Lf
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_Mb18E
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_n37rE
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_nFqfN
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_Nn5HO
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_NpjgZ
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_NsuYG
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:07 owamp_O3Fsh
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_ocz5q
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_OoMkr
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_P1YCC
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_Pa8Np
> drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:43 owamp_pbRb1
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:19 owamp_pCnw8
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_pliZ7
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_ptEtT
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_PTOGl
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_QkteT
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_QP8xX
> drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:52 owamp_qsQip
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_RdDjR
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_rdIpw
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_RinR3
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_RL0zw
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp__RQ57
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_s7lKI
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:05 owamp_SovaE
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_sqObj
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_Tp8ar
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_tT_gH
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_TZnhf
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_uCLJM
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_uhpRF
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_upwyY
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_VVOtV
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_wPbKx
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_XA5nh
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_xKcgv
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_xmZFB
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_Y0q75
> drwx------ 2 perfsonar perfsonar 4096 Mar 3 09:05 owamp_ykbDt
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_YPYl5
> drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_z9el1
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:40 owamp_ZlR4u
> drwx------ 2 perfsonar perfsonar 4096 Mar 4 08:39 owamp_zMQVW
>
>
>
>>>> On Mar 3, 2015, at 6:51 PM, Alan Whinery
>>>> <>
>>>> wrote:
>>>>
>>>> I have a host with pS-Performance Toolkit 3.4.1 (3.4.1-1.pSPS) with a
>>>> mix of powstream and traceroute tests running.
>>>>
>>>> Last Friday, at 23:21:12 I started getting a lot of messages like:
>>>>
>>>> 2015/02/26 23:21:12 (30583) DEBUG> DirQueue.pm:44
>>>> perfSONAR_PS::RegularTesting::DirQueue::worker_still_working -
>>>> worker_still_working called
>>>>
>>>> And the graphs stopped updating.
>>>>
>>>> I did find the line that logs the message in the source, but it's not
>>>> commented.
>>>>
>>>> So something is full, or stuck? Rebooting had no effect, disk has 858 GB
>>>> available...
>>>>
>>>> There were no instances of this log message prior to 2015/02/26
>>>> 23:21:12, grepping back through 7 days of logs.
>>>>
>


Archive powered by MHonArc 2.6.16.

Top of Page