perfSONAR User Q&A and Other Discussion

[Alan Whinery <>]

On 3/4/2015 8:40 AM, Alan Whinery wrote:
> On 3/4/2015 8:28 AM, Aaron Brown wrote:
>> Hey Alan,
>>
>> Are there a bunch of files in /var/lib/perfsonar/regular_testing?
>>
>> Cheers,
>> Aaron
Or perhaps I should have said directories...

"ls *" gets 125 directories, 54 regular files.

[root@uhmanoa-dl
 regular_testing]# ls -l
total 500
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:11 bwctl_0BQRw
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:17 bwctl_0QH8E
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:32 bwctl_5k0Wq
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:36 bwctl_71FtU
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 bwctl_AhVhc
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:07 bwctl_E9aEh
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:01 bwctl_EV9Z6
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:07 bwctl_EYlKB
drwx------ 2 perfsonar perfsonar 4096 Mar  3 08:58 bwctl_FlgvC
drwx------ 2 perfsonar perfsonar 4096 Feb 19 18:47 bwctl_gBppB
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:15 bwctl_kMDVi
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:38 bwctl_l4ksQ
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:07 bwctl_Mybgb
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:38 bwctl_QzaBA
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:34 bwctl_SAF_T
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:31 bwctl_TljeT
drwx------ 2 perfsonar perfsonar 4096 Feb 19 18:45 bwctl_u3Mum
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:36 bwctl_xOkeh
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:33 bwctl_XSDvn
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:16 bwctl_z4m5O
drwxr-x--- 4 perfsonar perfsonar 4096 Dec  9 13:11 esmond_latency_localhost
drwxr-x--- 4 perfsonar perfsonar 4096 Dec  9 13:11
esmond_throughput_localhost
drwxr-x--- 4 perfsonar perfsonar 4096 Dec  9 13:11
esmond_traceroute_localhost
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_1723R
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_2hu2R
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_3eKCy
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_3zqqB
drwx------ 2 perfsonar perfsonar 4096 Dec 24 13:59 owamp_46rTZ
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_5JObr
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_5x2ua
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_65dDj
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_6Jvkt
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_6TEyC
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_7jw_T
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_7qHIi
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9hu1N
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9nYtA
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_9rE5X
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9sD7v
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_9TSfJ
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_AKoMY
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:07 owamp_AYzch
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_AZRoN
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_bBAWm
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:07 owamp_BPHTO
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_bSKCn
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_cH9GH
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_ckzKy
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_CpoHI
drwx------ 2 perfsonar perfsonar 4096 Jan 13 02:00 owamp_DH3y4
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_D_nzz
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_E0uGT
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_e5Qff
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_f0BZ7
drwx------ 2 perfsonar perfsonar 4096 Jan 13 02:00 owamp_f8e2s
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_FEodt
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fFGwc
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fjTEN
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:07 owamp_fTuzd
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fuu1J
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_fUwhB
drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:47 owamp_Fvurb
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_fw0nn
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_GI0vV
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_GlieD
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_GO2E6
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_guJHF
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_hmofi
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_iaPC0
drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:41 owamp_j7HFV
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_jaidn
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_k680N
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_kHXyR
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_kSyX4
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_Ky6te
drwx------ 2 perfsonar perfsonar 4096 Dec 24 14:00 owamp_L3zmI
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_lBYgk
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_LkSyt
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_lLC2x
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_Lt422
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_lW5Lf
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_Mb18E
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_n37rE
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_nFqfN
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_Nn5HO
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_NpjgZ
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_NsuYG
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:07 owamp_O3Fsh
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_ocz5q
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_OoMkr
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_P1YCC
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_Pa8Np
drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:43 owamp_pbRb1
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:19 owamp_pCnw8
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_pliZ7
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_ptEtT
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_PTOGl
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_QkteT
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_QP8xX
drwx------ 2 perfsonar perfsonar 4096 Jan 20 07:52 owamp_qsQip
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_RdDjR
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_rdIpw
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_RinR3
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_RL0zw
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp__RQ57
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_s7lKI
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:05 owamp_SovaE
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_sqObj
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_Tp8ar
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_tT_gH
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_TZnhf
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_uCLJM
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_uhpRF
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_upwyY
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_VVOtV
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_wPbKx
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_XA5nh
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_xKcgv
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_xmZFB
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_Y0q75
drwx------ 2 perfsonar perfsonar 4096 Mar  3 09:05 owamp_ykbDt
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_YPYl5
drwx------ 2 perfsonar perfsonar 4096 Jan 12 12:20 owamp_z9el1
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:40 owamp_ZlR4u
drwx------ 2 perfsonar perfsonar 4096 Mar  4 08:39 owamp_zMQVW



>>> On Mar 3, 2015, at 6:51 PM, Alan Whinery 
>>> <>
>>>  wrote:
>>>
>>> I have a host with pS-Performance Toolkit    3.4.1 (3.4.1-1.pSPS) with a
>>> mix of powstream and traceroute tests running.
>>>
>>> Last Friday,  at 23:21:12 I started getting a lot of messages like:
>>>
>>>    2015/02/26 23:21:12 (30583) DEBUG> DirQueue.pm:44
>>> perfSONAR_PS::RegularTesting::DirQueue::worker_still_working -
>>> worker_still_working called
>>>
>>> And the graphs stopped updating.
>>>
>>> I did find the line that logs the message in the source, but it's not
>>> commented.
>>>
>>> So something is full, or stuck? Rebooting had no effect, disk has 858 GB
>>> available...
>>>
>>> There were no instances of this log message prior to 2015/02/26
>>> 23:21:12, grepping back through 7 days of logs.
>>>