perfSONAR User Q&A and Other Discussion

[Zhi-Wei Lu <>]

Hi Michael,

Sadly, it won't work for me!  I have deleted all the files in those 
directories, but once I started the regular testing, new files with "\\\..." 
start to populate these files! I guess that I need to have the new code 
before I can turn on regular testing.  It would fill 1 TB space in about 12 
hrs for this particular server.

Thank you for your suggestion.

Zhi-Wei Lu
IET-CR-Network Operations Center
University of California, Davis
(530) 752-0155

-----Original Message-----
From: Michael Johnson 
[mailto:]
 
Sent: Friday, December 19, 2014 8:22 AM
To: Zhi-Wei Lu
Cc: Aaron Brown; 

Subject: Re: [perfsonar-user] personar fills up the disk in 
esmond_latency_localhost and esmond_latency_traceroute directories

Hi,

Can you try editing the description? take all the backslashes out and also 
take any quote/apostrophe characters out of the description value. There's a 
bug that causes the backslashes to grow exponentially in some cases. I 
believe it's been fixed but not released yet.

Thanks,
Michael

On Fri, Dec 19, 2014 at 04:18:46PM +0000, Zhi-Wei Lu wrote:
>Hi Aaron,
>
>I removed those files, once I start regular_testing, the disk will be filled 
>soon.
>
>>            "address" : "64.124.161.242",
>>            "override_parameters" : {
>>               "type" : "bwtraceroute",
>>               "force_ipv4" : "1"
>>            },
>>            "description" : 
>> "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
>>
>> Tons of backslash, Do you know what process creates these *EMjQ files?
>>
>
>How could I get out of this self-destructive circle?
>
>Thank you.
>
>Zhi-Wei Lu
>IET-CR-Network Operations Center
>University of California, Davis
>(530) 752-0155
>
>-----Original Message-----
>From: Aaron Brown 
>[mailto:]
>Sent: Tuesday, December 16, 2014 10:40 AM
>To: Zhi-Wei Lu
>Cc: 
>
>Subject: Re: [perfsonar-user] personar fills up the disk in 
>esmond_latency_localhost and esmond_latency_traceroute directories
>
>Hi,
>
>The regular testing daemon creates those files whenever a test can’t be 
>written to esmond, so that it can be retried later. Note, this is different 
>from a test just failing. Whether a test fails or succeeds, something gets 
>written to esmond.
>
>You should be able to remove them without issue, albeit you’ll lose whatever 
>data might have been in there.
>
>Cheers,
>Aaron
>
>> On Dec 16, 2014, at 1:35 PM, Zhi-Wei Lu 
>> <>
>>  wrote:
>>
>> Thanks Jason.
>>
>> After I stopped the regular testing and  Cassandra, I still cannot use the 
>> "configure tests" at all,
>>
>> Gateway Time-out
>> The gateway did not receive a timely response from the upstream server or 
>> application.
>> Apache/2.2.15 (CentOS) Server at p51.noc.ucdavis.edu Port 443
>>
>> The files in
>> /var/lib/perfsonar/regular_testing/esmond_traceroute_localhost/failed/data/1/O
>> drwxrwxrwx 3 perfsonar perfsonar      4096 Dec 15 13:34 ..
>> -rw-rw-rw- 1 perfsonar perfsonar 536892791 Dec 16 09:32 
>> 50.20141216173234486173.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar      6265 Dec 16 09:32 
>> 50.20141216173247099266.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892828 Dec 16 09:33 
>> 50.20141216173323571178.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892869 Dec 16 09:33 
>> 50.20141216173353526886.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892819 Dec 16 09:33 
>> 50.20141216173355090130.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892842 Dec 16 09:35 
>> 50.20141216173536544323.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892805 Dec 16 09:35 
>> 50.20141216173539200618.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892906 Dec 16 09:37 
>> 50.20141216173749520175.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 805328325 Dec 16 09:37 
>> 50.20141216173752649319.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:39 
>> 50.20141216173957571983.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 805328326 Dec 16 09:40 
>> 50.20141216174025363413.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892843 Dec 16 09:40 
>> 50.20141216174045157625.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892879 Dec 16 09:44 
>> 50.20141216174410299038.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892862 Dec 16 09:44 
>> 50.20141216174425301435.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar      6265 Dec 16 09:44 
>> 50.20141216174456831046.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar      5817 Dec 16 09:44 
>> 50.20141216174459716445.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892915 Dec 16 09:46 
>> 50.20141216174603118253.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892831 Dec 16 09:46 
>> 50.20141216174631392221.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892831 Dec 16 09:46 
>> 50.20141216174633547370.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:48 
>> 50.20141216174841901446.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:49 
>> 50.20141216174912131552.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892888 Dec 16 09:49 
>> 50.20141216174933282328.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892888 Dec 16 09:49 
>> 50.20141216174937306445.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892904 Dec 16 09:51 
>> 50.20141216175133693082.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892904 Dec 16 09:52 
>> 50.20141216175219732395.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892859 Dec 16 09:52 
>> 50.20141216175235915105.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892859 Dec 16 09:54 
>> 50.20141216175410064202.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:54 
>> 50.20141216175411768689.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:54 
>> 50.20141216175438239629.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:56 
>> 50.20141216175617544033.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892825 Dec 16 09:57 
>> 50.20141216175659829503.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892825 Dec 16 09:58 
>> 50.20141216175822195902.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892825 Dec 16 09:58 
>> 50.20141216175825088636.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:59 
>> 50.20141216175901787207.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:59 
>> 50.20141216175908392282.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 10:00 
>> 50.20141216180048019864.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536893528 Dec 16 10:00 
>> 50.20141216180052676480.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536893528 Dec 16 10:02 
>> 50.20141216180234081082.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536893528 Dec 16 10:02 
>> 50.20141216180241641765.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892785 Dec 16 10:03 
>> 50.20141216180315834811.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892785 Dec 16 10:03 
>> 50.20141216180327139412.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar 536892785 Dec 16 10:05 
>> 50.20141216180517718112.EMjQ
>> -rw-rw-rw- 1 perfsonar perfsonar      1796 Dec 16 10:05 
>> 50.20141216180522445352.EMjQ
>>
>> It filled with
>> ...
>>            "address" : "64.124.161.242",
>>            "override_parameters" : {
>>               "type" : "bwtraceroute",
>>               "force_ipv4" : "1"
>>            },
>>            "description" : 
>> "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
>>
>> Tons of backslash, Do you know what process creates these *EMjQ files?
>>
>> Thank you.
>>
>> Zhi-Wei Lu
>> IET-CR-Network Operations Center
>> University of California, Davis
>> (530) 752-0155
>>
>> -----Original Message-----
>> From: Jason Zurawski 
>> [mailto:]
>> Sent: Monday, December 15, 2014 4:18 PM
>> To: Zhi-Wei Lu
>> Cc: 
>> 
>> Subject: Re: [perfsonar-user] personar fills up the disk in 
>> esmond_latency_localhost and esmond_latency_traceroute directories
>>
>> Hi Zhi-Wei;
>>
>> If you can get a terminal going, stop the regular testing daemon and 
>> cassandra:
>>
>>> sudo /etc/init.d/regular_testing stop
>>> sudo /etc/init.d/cassandra stop
>>
>> That should stop the ride long enough for you to make changes to the 
>> regular testing configuration.
>>
>> Thanks;
>>
>> -jason
>>
>> On Dec 15, 2014, at 5:03 PM, Zhi-Wei Lu 
>> <>
>>  wrote:
>>
>>> Hi all,
>>>
>>> We just noticed that one of our perfsonar box (3.4.2) was using a 
>>> temporary 192.168* address as its main address, thus all configured tests 
>>> failed a sometime.  I disabled the “internal testing” address and 
>>> restarted the system.  The system came back and it is busy generating 
>>> files in above directories ( */failed subdirectory).  Could someone 
>>> advise us how we can stop this downward spiral?
>>>
>>> I could not login into “Configure Tests” as the system is busing writing 
>>> disks and filling up the disk quickly.
>>>
>>> Thank you
>>>
>>> Zhi-Wei Lu
>>> IET-CR-Network Operations Center
>>> University of California, Davis
>>> (530) 752-0155
>

-- 
Michael Johnson
GlobalNOC Software Engineering
Indiana University

812-856-2771