perfSONAR User Q&A and Other Discussion

[Aaron Brown <>]

Hi Hector,

On Oct 8, 2014, at 1:43 PM, Hector Ordorica <> wrote:

Hi all, I'm experiencing the same issues Andrew is after the upgrade to the latest 3.4 release. In my case I have plenty of free space. I get the same error message:

Scheduled Tests Configuration Tool

Problem reading testing configuration: Problem reading Regular Testing Configuration: Problem reading Regular Testing configurat: Can't use string ("Line 4 malformed") as a HASH ref while "strict refs" in use at /opt/perfsonar_ps/toolkit/web/root/admin/regular_testing/../../../../lib/perfSONAR_PS/RegularTesting/Utils/SerializableObject.pm line 110.

The on-disk configuration has changed. Any changes you made have been lost.

sbin/service regular_testing restart

/etc/init.d/regular_testing stop: perfSONAR Regular Testing (pid 2474?) not running

waiting...

/opt/perfsonar_ps/regular_testing/bin/daemon --config=/opt/perfsonar_ps/regular_testing/etc/regular_testing.conf --pidfile=regular_testing.pid --piddir=/var/run --logger=/opt/perfsonar_ps/regular_testing/etc/regular_testing-logger.conf --user=perfsonar --group=perfsonar --daemonize

/etc/init.d/regular_testing start: perfSONAR Regular Testing could not be started

[root@thunderbird ~]# cat /var/log/perfsonar/regular_testing.log

2014/10/08 10:02:39 (2474) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: #

2014/10/08 10:02:39 (2474) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: test_result_directory

2014/10/08 10:02:39 (2474) ERROR> ConfigFile.pm:100 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Line 4 malformed

2014/10/08 10:02:39 (2474) ERROR> daemon:84 main:: - Problem parsing configuration file: Line 4 malformed

2014/10/08 10:32:23 (3495) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: #

2014/10/08 10:32:23 (3495) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: test_result_directory

2014/10/08 10:32:23 (3495) ERROR> ConfigFile.pm:100 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Line 4 malformed

2014/10/08 10:32:23 (3495) ERROR> daemon:84 main:: - Problem parsing configuration file: Line 4 malformed

Interestingly, this only failed on 1 of my 3 upgraded machines. They all started from the same patchlevel.

It looks like the regular testing software didn’t get configured as part of the Toolkit installation for some reason. Could you try running: “yum reinstall perl-perfSONAR_PS-Toolkit-SystemEnvironment”, and see if that works. If you could send us the output of the yum reinstall if it doesn’t work, that’d be helpful.

Cheer,s

Aaron

Thanks,

Hector Ordorica

UC San Diego

On Wed, Oct 8, 2014 at 5:38 AM, Andrew Lake <> wrote:

Hi Alex,

Glad you were able to track it down to a disk space issue. Between filled disks and all the reinstalls, its hard to say the exact state of things. Can you send me the following:

1. The output of /sbin/service regular_testing restart

2. The log file /var/log/perfsonar/regular_testing.log 

3. The file /opt/perfsonar_ps/regular_testing/etc/regular_testing.conf

Likely I'll just need to clean-up the third item by hand for you and send you a new version to drop in place, but the other two will help me verify that fact.

Thanks,

Andy

On Oct 8, 2014, at 12:21 AM, Alex Moura <> wrote:

Hello,

I verified that the host was very low on free disk space. I don't know how the upgrade did not failed completely.

I deleted old log files to free ~1.3GB on / and issued a reinstall of the packages with the string 'perfsonar' in its names, (while monitoring the disk space):

(Warning: data loss will occur when the commands below are executed).

$ rpm -qa | egrep -i '(perfsonar)' | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -srpm -qa | egrep -i perfsonar | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -s

Followed by a reinstallation of cassandra and http packages:

$ rpm -qa | egrep -i '(http|cassandra)' | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -s

The httpd reinstall failed. So it was needed to install manually:

$ sudo yum install -y httpd-2.2.15-31.el6.centos.x86_64

Then, I restarted both cassandra and httpd:

$ sudo service cassandra restart && sudo service httpd restart

This allowed to access the web gui again, but configurations were reset and all data was lost. 

At first I was unable to login in the "Administrative information" because of certificate problem. 

After a login in the root account the script asked to create a web admin account. That allowed to access the configuration in the web interface. The basic setup was reconfigured, but it seems the tests configurations scripts are still mangled:

https://200.133.192.50/toolkit/admin/regular_testing/

________________________________________________

Scheduled Tests Configuration Tool

Problem reading testing configuration: Problem reading Regular Testing Configuration: Problem reading Regular Testing configurat: Can't use string ("Line 4 malformed") as a HASH ref while "strict refs" in use at /opt/perfsonar_ps/toolkit/web/root/admin/regular_testing/../../../../lib/perfSONAR_PS/RegularTesting/Utils/SerializableObject.pm line 110.

The on-disk configuration has changed. Any changes you made have been lost.

________________________________________________

Maybe reinstalling some additional package will completely recover the host funcionalities?

Thanks,

Alex

On Tue, Oct 7, 2014 at 11:09 PM, Trey Dockendorf <> wrote:

I observed same.  'yum provides' shows it should exist.

- Trey

On Oct 7, 2014 9:02 PM, "Alex Moura" <> wrote:

On Tue, Oct 7, 2014 at 2:04 PM, Victor Lacort <> wrote:

Dear all,

Have you all tried to run /opt/perfsonar_ps/toolkit/scripts/discover_external_address after upgrading the machines?

It seems the boxes with version 3.4 does not have this script:

[ ~]$ sudo ls -lF /opt/perfsonar_ps/toolkit/scripts/discover_external_address

ls: unable to access /opt/perfsonar_ps/toolkit/scripts/discover_external_address: File or directory not found

-Alex

<regular_testing.conf>