perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi Hector,

I want to clarify a few things so we don't have anymore people repeat the same mistakes:

- Alex (not Andrew) encountered the install issues. He tried a "yum reinstall" approach after he saw the disk fail that I would not recommend anyone else repeat. It will cause more problems than it solves as a bunch of configuration files get overwritten if not done properly. In his situation with the disk, it was a reasonable thing  to do, but in general, "don't do that".

- It is intentional that you cannot login as root to the web interface and you are prompted at login to create a non-root user that can. We did this for security purposes. You can see this in our release notes and  upgrade notes: http://docs.perfsonar.net/manage_update.html#special-upgrade-notes. There's a lot of changes in this release so its easy to miss. 

- If you want to do a fresh install that's probably the easiest path forward. If not, aaron and I can continue to help you get things working. It is a little strange that the "SystemEnvironment" package was missing, and that seems to be the root of the problem if that's the case.

Thanks,

Andy

On Oct 8, 2014, at 2:31 PM, Hector Ordorica <> wrote:

Actually, that package was not installed at all. But even after installing it and a reboot, I still get the same error. I've even tried Andrew's method of removing all perfSonar packages with rpm and reinstalling with no luck. I don't really care to keep my old dat. The custom configs should gone, but that error remains.

At this point, I feel like I might just install from scratch from a fresh netinstall. It's technically one of 3 productions boxes on our campus. Thanks.

I just wanted to bring it up since everything Andrew pointed out happened to me as well. In each case, I started with a netinstall and was running the latest 3.3.x release before doing the upgrade to 3.4.

I also don't know if it was intentional, but on thunderbird (the broken machine) my root user permissions got screwed up somehow. Permissions got set to "/oot"

Perhaps when I got prompted to re-enter and create an admin account it screwed up root? Previously the root user was used to administer the interface. Now I cannot set it to root. I instead changed it to "admin" and now can administer the GUI.

This prompt is what I'm talking about:

[hordorica@falcon ~]$ sudo su -

[sudo] password for hordorica: 

There isn't a perfSONAR Website Administrator defined.    <---- I got this on all three machines after the upgrade.

Enter the user whose account you'd like to add. Just hit enter to exit:

Broken Machine:

[root@thunderbird etc]# ls -l

total 28

-rw-r--r-- 1 perfsonar perfsonar 2829 Oct  3 10:31 regular_testing.conf

-rw-r--r-- 1 perfsonar /oot 2828 Oct  8 10:40 regular_testing.conf-1

-rw-r--r-- 1 perfsonar /oot 2828 Oct  8 10:40 regular_testing.conf-2

-rw-r--r-- 1 root      /oot      2829 Oct  8 10:51 regular_testing.conf-3

Good machine:

[root@merlin etc]# ls -l

total 20

-rw-r--r-- 1 root      root      6174 Oct  7 08:23 regular_testing.conf

-rw-r--r-- 1 root      root      2802 Oct  7 08:23 regular_testing.conf-1

-rw-r--r-- 1 perfsonar perfsonar 2829 Oct  3 10:31 regular_testing.conf.install.back

-rw-r--r-- 1 perfsonar perfsonar  742 Oct  3 10:31 regular_testing-logger.conf

On Wed, Oct 8, 2014 at 11:04 AM, Aaron Brown <> wrote:

Hi Hector,

On Oct 8, 2014, at 1:43 PM, Hector Ordorica <> wrote:

Hi all, I'm experiencing the same issues Andrew is after the upgrade to the latest 3.4 release. In my case I have plenty of free space. I get the same error message:

Scheduled Tests Configuration Tool

Problem reading testing configuration: Problem reading Regular Testing Configuration: Problem reading Regular Testing configurat: Can't use string ("Line 4 malformed") as a HASH ref while "strict refs" in use at /opt/perfsonar_ps/toolkit/web/root/admin/regular_testing/../../../../lib/perfSONAR_PS/RegularTesting/Utils/SerializableObject.pm line 110.

The on-disk configuration has changed. Any changes you made have been lost.

sbin/service regular_testing restart

/etc/init.d/regular_testing stop: perfSONAR Regular Testing (pid 2474?) not running

waiting...

/opt/perfsonar_ps/regular_testing/bin/daemon --config=/opt/perfsonar_ps/regular_testing/etc/regular_testing.conf --pidfile=regular_testing.pid --piddir=/var/run --logger=/opt/perfsonar_ps/regular_testing/etc/regular_testing-logger.conf --user=perfsonar --group=perfsonar --daemonize

/etc/init.d/regular_testing start: perfSONAR Regular Testing could not be started

[root@thunderbird ~]# cat /var/log/perfsonar/regular_testing.log

2014/10/08 10:02:39 (2474) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: #

2014/10/08 10:02:39 (2474) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: test_result_directory

2014/10/08 10:02:39 (2474) ERROR> ConfigFile.pm:100 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Line 4 malformed

2014/10/08 10:02:39 (2474) ERROR> daemon:84 main:: - Problem parsing configuration file: Line 4 malformed

2014/10/08 10:32:23 (3495) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: #

2014/10/08 10:32:23 (3495) DEBUG> ConfigFile.pm:108 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Setting variable: test_result_directory

2014/10/08 10:32:23 (3495) ERROR> ConfigFile.pm:100 perfSONAR_PS::RegularTesting::Utils::ConfigFile::parse_file - Line 4 malformed

2014/10/08 10:32:23 (3495) ERROR> daemon:84 main:: - Problem parsing configuration file: Line 4 malformed

Interestingly, this only failed on 1 of my 3 upgraded machines. They all started from the same patchlevel.

It looks like the regular testing software didn’t get configured as part of the Toolkit installation for some reason. Could you try running: “yum reinstall perl-perfSONAR_PS-Toolkit-SystemEnvironment”, and see if that works. If you could send us the output of the yum reinstall if it doesn’t work, that’d be helpful.

Cheer,s

Aaron

Thanks,

Hector Ordorica

UC San Diego

On Wed, Oct 8, 2014 at 5:38 AM, Andrew Lake <> wrote:

Hi Alex,

Glad you were able to track it down to a disk space issue. Between filled disks and all the reinstalls, its hard to say the exact state of things. Can you send me the following:

1. The output of /sbin/service regular_testing restart

2. The log file /var/log/perfsonar/regular_testing.log 

3. The file /opt/perfsonar_ps/regular_testing/etc/regular_testing.conf

Likely I'll just need to clean-up the third item by hand for you and send you a new version to drop in place, but the other two will help me verify that fact.

Thanks,

Andy

On Oct 8, 2014, at 12:21 AM, Alex Moura <> wrote:

Hello,

I verified that the host was very low on free disk space. I don't know how the upgrade did not failed completely.

I deleted old log files to free ~1.3GB on / and issued a reinstall of the packages with the string 'perfsonar' in its names, (while monitoring the disk space):

(Warning: data loss will occur when the commands below are executed).

$ rpm -qa | egrep -i '(perfsonar)' | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -srpm -qa | egrep -i perfsonar | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -s

Followed by a reinstallation of cassandra and http packages:

$ rpm -qa | egrep -i '(http|cassandra)' | xargs -n1 -I '{}' echo "yum -y remove '{}' && yum -y install '{}'" | sudo -s

The httpd reinstall failed. So it was needed to install manually:

$ sudo yum install -y httpd-2.2.15-31.el6.centos.x86_64

Then, I restarted both cassandra and httpd:

$ sudo service cassandra restart && sudo service httpd restart

This allowed to access the web gui again, but configurations were reset and all data was lost. 

At first I was unable to login in the "Administrative information" because of certificate problem. 

After a login in the root account the script asked to create a web admin account. That allowed to access the configuration in the web interface. The basic setup was reconfigured, but it seems the tests configurations scripts are still mangled:

https://200.133.192.50/toolkit/admin/regular_testing/

________________________________________________

Scheduled Tests Configuration Tool

Problem reading testing configuration: Problem reading Regular Testing Configuration: Problem reading Regular Testing configurat: Can't use string ("Line 4 malformed") as a HASH ref while "strict refs" in use at /opt/perfsonar_ps/toolkit/web/root/admin/regular_testing/../../../../lib/perfSONAR_PS/RegularTesting/Utils/SerializableObject.pm line 110.

The on-disk configuration has changed. Any changes you made have been lost.

________________________________________________

Maybe reinstalling some additional package will completely recover the host funcionalities?

Thanks,

Alex

On Tue, Oct 7, 2014 at 11:09 PM, Trey Dockendorf <> wrote:

I observed same.  'yum provides' shows it should exist.

- Trey

On Oct 7, 2014 9:02 PM, "Alex Moura" <> wrote:

On Tue, Oct 7, 2014 at 2:04 PM, Victor Lacort <> wrote:

Dear all,

Have you all tried to run /opt/perfsonar_ps/toolkit/scripts/discover_external_address after upgrading the machines?

It seems the boxes with version 3.4 does not have this script:

[ ~]$ sudo ls -lF /opt/perfsonar_ps/toolkit/scripts/discover_external_address

ls: unable to access /opt/perfsonar_ps/toolkit/scripts/discover_external_address: File or directory not found

-Alex

<regular_testing.conf>

<perfsonar_broke.png>