perfSONAR User Q&A and Other Discussion

[Michael Johnson <>]

Hi Zhi-Wei,

I'm glad you were able to fix it! When I asked you to edit the description, regular_testing.conf is what I had in mind -- sorry I was unclear. 

We have a fix for this bug, but it hasn't been released yet. In the meantime, 
avoid using quotes or backslashes in your descriptions and you shouldn't have 
any problems.

I see you figured out how to start NDT as well. Let us know if you need 
anything else.

Thanks,
Michael

On Tue, Dec 23, 2014 at 11:08:19PM +0000, Zhi-Wei Lu wrote:
> Hi Michael,
>
> I think that the problem is probably solved.  As I was trying to gather the 
> file:
> /opt/perfsonar_ps/regular_testing/etc/regular_testing.conf
>
> I noticed that this file is 500 +MB in size, one of the description field was filled with the 
> dreaded "\\\\...", after I used sed to remove the offending character, I restarted the 
> regular_testing again.  It appears to run just fine.  I will keep my eye on it and I was able to 
> get into the "Configure Tests" Tab.
>
> I don't quite know how this configuration file got corrupted.  Thanks again for leading me to the 
> "debug".  By the way, do you know which command I should run to start the 
> "stopped" Network Diagnostic Tester (NDT)   Not Running"?
>
> Have a great Holiday Season.
>
> Zhi-Wei Lu
> IET-CR-Network Operations Center
> University of California, Davis
> (530) 752-0155
>
> -----Original Message-----
> From: 
>
>  
> [mailto:]
>  On Behalf Of Michael Johnson
> Sent: Tuesday, December 23, 2014 12:10 PM
> To: Zhi-Wei Lu
> Cc: 
>
> Subject: Re: [perfsonar-user] personar fills up the disk in 
> esmond_latency_localhost and esmond_latency_traceroute directories
>
> Hi Zhi-Wei,
>
> Has this been resolved? If not, can you please send
>
> /opt/perfsonar_ps/regular_testing/etc/regular_testing.conf
>
> You'll want to remove passwords from the file, or send it directly to me.
>
> Thanks,
> Michael
>
> On Fri, Dec 19, 2014 at 04:18:46PM +0000, Zhi-Wei Lu wrote:
> > Hi Aaron,
> >
> > I removed those files, once I start regular_testing, the disk will be filled 
> > soon.
> >
> > >            "address" : "64.124.161.242",
> > >            "override_parameters" : {
> > >               "type" : "bwtraceroute",
> > >               "force_ipv4" : "1"
> > >            },
> > >            "description" : 
> > > "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
> > >
> > > Tons of backslash, Do you know what process creates these *EMjQ files?
> >
> > How could I get out of this self-destructive circle?
> >
> > Thank you.
> >
> > Zhi-Wei Lu
> > IET-CR-Network Operations Center
> > University of California, Davis
> > (530) 752-0155
> >
> > -----Original Message-----
> > From: Aaron Brown 
> > [mailto:]
> > Sent: Tuesday, December 16, 2014 10:40 AM
> > To: Zhi-Wei Lu
> > Cc: 
> >
> > Subject: Re: [perfsonar-user] personar fills up the disk in 
> > esmond_latency_localhost and esmond_latency_traceroute directories
> >
> > Hi,
> >
> > The regular testing daemon creates those files whenever a test can’t be 
> > written to esmond, so that it can be retried later. Note, this is different 
> > from a test just failing. Whether a test fails or succeeds, something gets 
> > written to esmond.
> >
> > You should be able to remove them without issue, albeit you’ll lose whatever 
> > data might have been in there.
> >
> > Cheers,
> > Aaron
> >
> > > On Dec 16, 2014, at 1:35 PM, Zhi-Wei Lu 
> > > <>
> > >  wrote:
> > >
> > > Thanks Jason.
> > >
> > > After I stopped the regular testing and  Cassandra, I still cannot use the 
> > > "configure tests" at all,
> > >
> > > Gateway Time-out
> > > The gateway did not receive a timely response from the upstream server or 
> > > application.
> > > Apache/2.2.15 (CentOS) Server at p51.noc.ucdavis.edu Port 443
> > >
> > > The files in
> > > /var/lib/perfsonar/regular_testing/esmond_traceroute_localhost/failed/data/1/O
> > > drwxrwxrwx 3 perfsonar perfsonar      4096 Dec 15 13:34 ..
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892791 Dec 16 09:32 
> > > 50.20141216173234486173.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar      6265 Dec 16 09:32 
> > > 50.20141216173247099266.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892828 Dec 16 09:33 
> > > 50.20141216173323571178.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892869 Dec 16 09:33 
> > > 50.20141216173353526886.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892819 Dec 16 09:33 
> > > 50.20141216173355090130.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892842 Dec 16 09:35 
> > > 50.20141216173536544323.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892805 Dec 16 09:35 
> > > 50.20141216173539200618.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892906 Dec 16 09:37 
> > > 50.20141216173749520175.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 805328325 Dec 16 09:37 
> > > 50.20141216173752649319.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:39 
> > > 50.20141216173957571983.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 805328326 Dec 16 09:40 
> > > 50.20141216174025363413.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892843 Dec 16 09:40 
> > > 50.20141216174045157625.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892879 Dec 16 09:44 
> > > 50.20141216174410299038.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892862 Dec 16 09:44 
> > > 50.20141216174425301435.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar      6265 Dec 16 09:44 
> > > 50.20141216174456831046.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar      5817 Dec 16 09:44 
> > > 50.20141216174459716445.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892915 Dec 16 09:46 
> > > 50.20141216174603118253.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892831 Dec 16 09:46 
> > > 50.20141216174631392221.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892831 Dec 16 09:46 
> > > 50.20141216174633547370.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:48 
> > > 50.20141216174841901446.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:49 
> > > 50.20141216174912131552.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892888 Dec 16 09:49 
> > > 50.20141216174933282328.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892888 Dec 16 09:49 
> > > 50.20141216174937306445.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892904 Dec 16 09:51 
> > > 50.20141216175133693082.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892904 Dec 16 09:52 
> > > 50.20141216175219732395.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892859 Dec 16 09:52 
> > > 50.20141216175235915105.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892859 Dec 16 09:54 
> > > 50.20141216175410064202.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:54 
> > > 50.20141216175411768689.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:54 
> > > 50.20141216175438239629.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892852 Dec 16 09:56 
> > > 50.20141216175617544033.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892825 Dec 16 09:57 
> > > 50.20141216175659829503.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892825 Dec 16 09:58 
> > > 50.20141216175822195902.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892825 Dec 16 09:58 
> > > 50.20141216175825088636.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:59 
> > > 50.20141216175901787207.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 09:59 
> > > 50.20141216175908392282.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892864 Dec 16 10:00 
> > > 50.20141216180048019864.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536893528 Dec 16 10:00 
> > > 50.20141216180052676480.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536893528 Dec 16 10:02 
> > > 50.20141216180234081082.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536893528 Dec 16 10:02 
> > > 50.20141216180241641765.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892785 Dec 16 10:03 
> > > 50.20141216180315834811.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892785 Dec 16 10:03 
> > > 50.20141216180327139412.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar 536892785 Dec 16 10:05 
> > > 50.20141216180517718112.EMjQ
> > > -rw-rw-rw- 1 perfsonar perfsonar      1796 Dec 16 10:05 
> > > 50.20141216180522445352.EMjQ
> > >
> > > It filled with
> > > ...
> > >            "address" : "64.124.161.242",
> > >            "override_parameters" : {
> > >               "type" : "bwtraceroute",
> > >               "force_ipv4" : "1"
> > >            },
> > >            "description" : 
> > > "\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
> > >
> > > Tons of backslash, Do you know what process creates these *EMjQ files?
> > >
> > > Thank you.
> > >
> > > Zhi-Wei Lu
> > > IET-CR-Network Operations Center
> > > University of California, Davis
> > > (530) 752-0155
> > >
> > > -----Original Message-----
> > > From: Jason Zurawski 
> > > [mailto:]
> > > Sent: Monday, December 15, 2014 4:18 PM
> > > To: Zhi-Wei Lu
> > > Cc: 
> > >
> > > Subject: Re: [perfsonar-user] personar fills up the disk in 
> > > esmond_latency_localhost and esmond_latency_traceroute directories
> > >
> > > Hi Zhi-Wei;
> > >
> > > If you can get a terminal going, stop the regular testing daemon and 
> > > cassandra:
> > >
> > > > sudo /etc/init.d/regular_testing stop
> > > > sudo /etc/init.d/cassandra stop
> > >
> > > That should stop the ride long enough for you to make changes to the regular 
> > > testing configuration.
> > >
> > > Thanks;
> > >
> > > -jason
> > >
> > > On Dec 15, 2014, at 5:03 PM, Zhi-Wei Lu 
> > > <>
> > >  wrote:
> > >
> > > > Hi all,
> > > >
> > > > We just noticed that one of our perfsonar box (3.4.2) was using a temporary 
> > > > 192.168* address as its main address, thus all configured tests failed a 
> > > > sometime.  I disabled the “internal testing” address and restarted the 
> > > > system.  The system came back and it is busy generating files in above 
> > > > directories ( */failed subdirectory).  Could someone advise us how we can 
> > > > stop this downward spiral?
> > > >
> > > > I could not login into “Configure Tests” as the system is busing writing 
> > > > disks and filling up the disk quickly.
> > > >
> > > > Thank you
> > > >
> > > > Zhi-Wei Lu
> > > > IET-CR-Network Operations Center
> > > > University of California, Davis
> > > > (530) 752-0155
>
> --
> Michael Johnson
> GlobalNOC Software Engineering
> Indiana University
>
> 812-856-2771

--
Michael Johnson
GlobalNOC Software Engineering
Indiana University

812-856-2771

Attachment: pgp1SIL8mrSD2.pgp
Description: PGP signature