perfSONAR User Q&A and Other Discussion

[Pedro Reis <>]

Hello all,

I would like to report a similar a similar situation.

I have 2 toolkit that are sending the measurements to a single Archive.
Now last Saturday around 8AM the MA had a slight problem and stopped
processing information (I had to re-set the python environment again).

Since I fixed the MA the two toolkits have the the regulartesting.log
full of messages like this:
2016/12/12 15:11:02 (42410) WARN> regulartesting.pl:103 main::__ANON__ -
Warned: IPC::DirQueue: killed stale lockfile:
/var/lib/perfsonar/regulartesting/esmond_latency_<MA_FQDN_HERE>/active/active/50.20161212035623328938.EMjQ4Mw
at /usr/share/perl5/IPC/DirQueue.pm line 519.

And the regulartesting process is using up almost all the CPU
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1425 perfsona  20   0  237m 112m 1244 R 99.8  0.7   2:29.82 regulartesting.
  935 cassandr  20   0 21.6g 1.4g  16m S 29.0  9.4   3:20.16 java
 2505 apache    20   0  673m  46m 4712 S 11.2  0.3   0:03.48 httpd


I'm hopping the regular testing is just processing the data from the
failed connections/measurements from when the MA was down.
Because I tried to see other errors/problem in other logs and didn't
seem to find anything relevant!

Until this date I'm still not seeing any new measurements at the MA :(

Com meus melhores cumprimentos | Best Regards
Pedro Reis
Área de Serviços de Rede | Network Services Area
FCT|FCCN
Av. do Brasil, n.º 101
1700-066 Lisboa - Portugal
Telefone|Phone +351 218 440 100; Fax +351 218 472 167
www.fccn.pt

On 2016/06/19 18:15, Lixin Liu wrote:
> It appears the problem is gone, not sure how but tests results are 
> available again.
> 
> Thanks,
> 
> Lixin.
> 
> On 2016-06-18, 10:14 PM, "Lixin Liu" 
> <
>  on behalf of 
> >
>  wrote:
> 
> Hi,
> 
> One of my latency hosts stopped reporting test results starting sometime 
> early today. 
> I see the load on the process
> 
>       perfSONAR Regular Testing: Measurement Archive: 
> esmond_latency_localhost
> 
> is always at 100% and regulartesting.log continues showing errors like this:
> 
> 2016/06/18 21:48:23 (2656) WARN> regulartesting.pl:103 main::__ANON__ - 
> Warned: IPC::DirQueue: killed stale lockfile: 
> /var/lib/perfsonar/regulartesting/esmond_latency_localhost/active/active/50.20160618092329343238.EMjI4OQ
>  at /usr/share/perl5/IPC/DirQueue.pm line 519.
> 2016/06/18 21:48:24 (2656) WARN> regulartesting.pl:103 main::__ANON__ - 
> Warned: IPC::DirQueue: killed stale lockfile: 
> /var/lib/perfsonar/regulartesting/esmond_latency_localhost/active/active/50.20160618092329700337.EMjI5MQ
>  at /usr/share/perl5/IPC/DirQueue.pm line 519.
> 
> I hope someone could help me to figure out what needs to be done to resolve 
> this
> problem. The hostname of the machine is lat-usask.westgrid.ca.
> 
> Thanks,
> 
> Lixin Liu
> Compute Canada & WestGrid
> 
> 
> 
> 
> 
> 
>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature