perfSONAR User Q&A and Other Discussion

[Hector Ordorica <>]

Hi all, I'm experiencing the same issues Andrew is after the upgrade to the latest 3.4 release. In my case I have plenty of free space. I get the same error message:

Scheduled Tests Configuration Tool

Problem reading testing configuration: Problem reading Regular Testing Configuration: Problem reading Regular Testing configurat: Can't use string ("Line 4 malformed") as a HASH ref while "strict refs" in use at /opt/perfsonar_ps/toolkit/web/root/admin/regular_testing/../../../../lib/perfSONAR_PS/RegularTesting/Utils/SerializableObject.pm line 110.

The on-disk configuration has changed. Any changes you made have been lost.

sbin/service regular_testing restart

/etc/init.d/regular_testing stop: perfSONAR Regular Testing (pid 2474?) not running

waiting...

/opt/perfsonar_ps/regular_testing/bin/daemon --config=/opt/perfsonar_ps/regular_testing/etc/regular_testing.conf --pidfile=regular_testing.pid --piddir=/var/run --logger=/opt/perfsonar_ps/regular_testing/etc/regular_testing-logger.conf --user=perfsonar --group=perfsonar --daemonize

/etc/init.d/regular_testing start: perfSONAR Regular Testing could not be started

[root@thunderbird ~]# cat /var/log/perfsonar/regular_testing.log

2014/10/08 10:02:39 (2474) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: #

2014/10/08 10:02:39 (2474) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: test_result_directory

2014/10/08 10:02:39 (2474) ERROR> ConfigFile.pm:100 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Line 4 malformed

2014/10/08 10:02:39 (2474) ERROR> daemon:84 main:: - Problem parsing configuration file: Line 4 malformed

2014/10/08 10:32:23 (3495) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: #

2014/10/08 10:32:23 (3495) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: test_result_directory

2014/10/08 10:32:23 (3495) ERROR> ConfigFile.pm:100 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Line 4 malformed

2014/10/08 10:32:23 (3495) ERROR> daemon:84 main:: - Problem parsing configuration file: Line 4 malformed

Interestingly, this only failed on 1 of my 3 upgraded machines. They all started from the same patchlevel.

Thanks,

Hector Ordorica

UC San Diego

On Wed, Oct 8, 2014 at 5:38 AM, Andrew Lake <> wrote:

Hi Alex,

Glad you were able to track it down to a disk space issue. Between filled disks and all the reinstalls, its hard to say the exact state of things. Can you send me the following:

1. The output of /sbin/service regular_testing restart

2. The log file /var/log/perfsonar/regular_testing.log 

3. The file /opt/perfsonar_ps/regular_testing/etc/regular_testing.conf

Likely I'll just need to clean-up the third item by hand for you and send you a new version to drop in place, but the other two will help me verify that fact.

Thanks,

Andy

On Oct 8, 2014, at 12:21 AM, Alex Moura <> wrote:

Hello,

I verified that the host was very low on free disk space. I don't know how the upgrade did not failed completely.

I deleted old log files to free ~1.3GB on / and issued a reinstall of the packages with the string 'perfsonar' in its names, (while monitoring the disk space):

(Warning: data loss will occur when the commands below are executed).

$ rpm -qa | egrep -i '(perfsonar)' | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -srpm -qa | egrep -i perfsonar | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -s

Followed by a reinstallation of cassandra and http packages:

$ rpm -qa | egrep -i '(http|cassandra)' | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -s

The httpd reinstall failed. So it was needed to install manually:

$ sudo yum install -y httpd-2.2.15-31.el6.centos.x86_64

Then, I restarted both cassandra and httpd:

$ sudo service cassandra restart && sudo service httpd restart

This allowed to access the web gui again, but configurations were reset and all data was lost. 

At first I was unable to login in the "Administrative information" because of certificate problem. 

After a login in the root account the script asked to create a web admin account. That allowed to access the configuration in the web interface. The basic setup was reconfigured, but it seems the tests configurations scripts are still mangled:

https://200.133.192.50/toolkit/admin/regular_testing/

________________________________________________

Scheduled Tests Configuration Tool

Problem reading testing configuration: Problem reading Regular Testing Configuration: Problem reading Regular Testing configurat: Can't use string ("Line 4 malformed") as a HASH ref while "strict refs" in use at /opt/perfsonar_ps/toolkit/web/root/admin/regular_testing/../../../../lib/perfSONAR_PS/RegularTesting/Utils/SerializableObject.pm line 110.

The on-disk configuration has changed. Any changes you made have been lost.

________________________________________________

Maybe reinstalling some additional package will completely recover the host funcionalities?

Thanks,

Alex

On Tue, Oct 7, 2014 at 11:09 PM, Trey Dockendorf <> wrote:

I observed same.  'yum provides' shows it should exist.

- Trey

On Oct 7, 2014 9:02 PM, "Alex Moura" <> wrote:

On Tue, Oct 7, 2014 at 2:04 PM, Victor Lacort <> wrote:

Dear all,

Have you all tried to run /opt/perfsonar_ps/toolkit/scripts/discover_external_address after upgrading the machines?

It seems the boxes with version 3.4 does not have this script:

[ ~]$ sudo ls -lF /opt/perfsonar_ps/toolkit/scripts/discover_external_address

ls: unable to access /opt/perfsonar_ps/toolkit/scripts/discover_external_address: File or directory not found

-Alex

Attachment: regular_testing.conf
Description: Binary data