perfSONAR User Q&A and Other Discussion

["Garnizov, Ivan (RRZE)" <>]

Hello Pedro,

 

Unfortunately this is somewhat expected. Once the stalled information expires, there are no easy ways of recollecting it (not sure even if there are).

Please stand by. Next pS release applies a different approach that is also looking at improving/fixing the problem.

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

 

 

-----Original Message-----
From: Pedro Reis []
Sent: Mittwoch, 14. Dezember 2016 09:29
To: Garnizov, Ivan (RRZE);
Subject: Re: [perfsonar-user] perfSONAR host stop report the test results

 

Hello Ivan, All,

 

Thanks for the info.

The system eventually recovered after a few hours processing the stale

lockfiles.

 

Now, I was expecting to see the results in the MA, but I'm getting a big

blank. In both toolkits the readings are there, but looks like they

didn't manage to (re)send it to the MA, or the MA didn't processed the

information correctly :(

 

Com meus melhores cumprimentos | Best Regards

Pedro Reis

Área de Serviços de Rede | Network Services Area

FCT|FCCN

Av. do Brasil, n.º 101

1700-066 Lisboa - Portugal

Telefone|Phone +351 218 440 100; Fax +351 218 472 167

www.fccn.pt

 

On 2016/12/13 09:53, Garnizov, Ivan (RRZE) wrote:

> Hi Pedro,

> 

> Generally all the scheduled tests on the toolkit generate files in that

> /var/lib/perfsonar/regulartesting/ folder.

> I would suggest to first suspend the regulartesting service for a while

> and try to stop all the active processes running which try to write in

> that folder and (optionally) clean after them.

> A very good option for you could be to just restart the server, but make

> note of the currently created folders in /var/lib/perfsonar/regulartesting/

> You might want to clean those afterwards. You will not have easy options

> to do this after the restart.

> 

> Regards,

> Ivan Garnizov

> 

> /GEANT SA1T2: pS deployments GN Operations/

> /GEANT SA2T3: pS development team/

> /GEANT SA3T5: eduPERT team/

> 

> 

> 

> -----Original Message-----

> From:

> [] On Behalf Of Pedro Reis

> Sent: Montag, 12. Dezember 2016 16:38

> To:

> Subject: Re: [perfsonar-user] perfSONAR host stop report the test results

> 

> Hello all,

> 

> I would like to report a similar a similar situation.

> 

> I have 2 toolkit that are sending the measurements to a single Archive.

> Now last Saturday around 8AM the MA had a slight problem and stopped

> processing information (I had to re-set the python environment again).

> 

> Since I fixed the MA the two toolkits have the the regulartesting.log

> full of messages like this:

> 2016/12/12 15:11:02 (42410) WARN> regulartesting.pl:103 main::__ANON__ -

> Warned: IPC::DirQueue: killed stale lockfile:

> /var/lib/perfsonar/regulartesting/esmond_latency_<MA_FQDN_HERE>/active/active/50.20161212035623328938.EMjQ4Mw

> at /usr/share/perl5/IPC/DirQueue.pm line 519.

> 

> And the regulartesting process is using up almost all the CPU

>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

> 1425 perfsona  20   0  237m 112m 1244 R 99.8  0.7   2:29.82 regulartesting.

>   935 cassandr  20   0 21.6g 1.4g  16m S 29.0  9.4   3:20.16 java

> 2505 apache    20   0  673m  46m 4712 S 11.2  0.3   0:03.48 httpd

> 

> 

> I'm hopping the regular testing is just processing the data from the

> failed connections/measurements from when the MA was down.

> Because I tried to see other errors/problem in other logs and didn't

> seem to find anything relevant!

> 

> Until this date I'm still not seeing any new measurements at the MA :(

> 

> Com meus melhores cumprimentos | Best Regards

> Pedro Reis

> Área de Serviços de Rede | Network Services Area

> FCT|FCCN

> Av. do Brasil, n.º 101

> 1700-066 Lisboa - Portugal

> Telefone|Phone +351 218 440 100; Fax +351 218 472 167

> www.fccn.pt <http://www.fccn.pt>

> 

> On 2016/06/19 18:15, Lixin Liu wrote:

>> It appears the problem is gone, not sure how but tests results are available again.

>>

>> Thanks,

>>

>> Lixin.

>>

>> On 2016-06-18, 10:14 PM, "Lixin Liu" < on behalf of

> <>>

> wrote:

>>

>> Hi,

>>

>> One of my latency hosts stopped reporting test results starting sometime early today.

>> I see the load on the process

>>

>>        perfSONAR Regular Testing: Measurement Archive: esmond_latency_localhost

>>

>> is always at 100% and regulartesting.log continues showing errors like this:

>>

>> 2016/06/18 21:48:23 (2656) WARN> regulartesting.pl:103 main::__ANON__ - Warned: IPC::DirQueue: killed stale lockfile: /var/lib/perfsonar/regulartesting/esmond_latency_localhost/active/active/50.20160618092329343238.EMjI4OQ at /usr/share/perl5/IPC/DirQueue.pm line 519.

>> 2016/06/18 21:48:24 (2656) WARN> regulartesting.pl:103 main::__ANON__ - Warned: IPC::DirQueue: killed stale lockfile: /var/lib/perfsonar/regulartesting/esmond_latency_localhost/active/active/50.20160618092329700337.EMjI5MQ at /usr/share/perl5/IPC/DirQueue.pm line 519.

>>

>> I hope someone could help me to figure out what needs to be done to resolve this

>> problem. The hostname of the machine is lat-usask.westgrid.ca.

>>

>> Thanks,

>>

>> Lixin Liu

>> Compute Canada & WestGrid

>>

>>

>>

>>

>>

>>

>>

> 

>