perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi Ryan,

Does the page load now? I wonder if the hanging was caused by some data center work at ESnet which hosts a file the page loads to determine if your host is in the lookup service. It should timeout but it not doing so has been a common complaint recently. The file in questions should be available again. FWIW, I am able to get your non-auth page to load without problem and see two tests listed.

Thanks,

Andy

On October 9, 2017 at 11:20:19 AM, Decker, Ryan C. () wrote:

Hi Andy,

I have since reinstalled the server in an attempt to bring it back online. I have expanded the tar and i see both a postgresql_data directory with dump files and a cassandra_data directory with many sub directories. There is an "esmond_snapshot" directory with db files. When i did the backup i included --data and i also included --data when i attempted to restore this on Friday.

I have reinstalled the server using CentOS7 and attempted to restore the file without the --data flag. I seemed to have better luck this time but the main toolkit page is hanging with errors like this in ssl_error_log:

[Mon Oct 09 11:14:43.115939 2017] [cgi:warn] [pid 2486] [client 199.223.240.10:38530] AH01220: Timeout waiting for output from CGI script /usr/lib/perfsonar/web-ng/root/services/host.cgi, referer: https://ps1.siena.edu/toolkit/auth/
[Mon Oct 09 11:14:43.115969 2017] [cgi:error] [pid 2486] [client 199.223.240.10:38530] Script timed out before returning headers: host.cgi, referer: https://ps1.siena.edu/toolkit/auth/

I have the hosts file configured properly. I don't care much for the data at this point i just would like to get this server back online.

On Mon, Oct 9, 2017 at 11:08 AM, Andrew Lake <> wrote:

Hi Ryan,

Looking at your measurement archive I don't see any results, new or old: https://66.194.180.9/esmond/perfsonar/archive/

For the old results, when you originally did the back-up did you include —data? or did you only do —data when you did the restore? If you want to see if you have the data, you can manually expand the tarball (e.g. tar -xzf <file.tar>) and in the folder that gets created look for a cassandra_data directory and a postgresql_data directory. Also verify that they are both relatively large in file size. How big depends on how much old data you had, but if its 0 bytes or a few KB probably something did not get copied properly. If there a few MB/GB or bigger that means things likely worked but for some reason did not get restored properly.

Also, as for why new tasks are failing, for some reason your server stopped letting me connect after I looked at the archive so couldn’t dig out exact error (perhaps you are working on it?). Either in the diags command you ran or in /var/log/pscheduler/pscheduler.log I’d expect to see some type of error. If not, you may want to look at /var/log/perfsonar/meshconfig-agent.log which is responsible for creating the tasks to see if its having trouble creating the tasks.

Thanks,

Andy

On October 9, 2017 at 9:14:05 AM, Decker, Ryan C. () wrote:

Hey Ivan,

The backup script said it succeeded. Is there a way to validate the backup file that the script produces?

On Mon, Oct 9, 2017 at 9:03 AM, Garnizov, Ivan (RRZE) <> wrote:

Hello Ryan,

 

Please note that your pscheduler call (shared in your email) retrieves these results from the volatile scheduler DB and not from the permanent measurement archive = Esmond.

The graphs are being displayed only based on the results stored in Esmond.

Are you sure your backup procedure succeeded?

Are the results from your measurements updated after the upgrade? Or are you also missing the results after the upgrade?

 

Regards,

Ivan Garnizov

 

GEANT SA1T2: pS deployments GN Operations

GEANT SA2T3: pS development team

GEANT SA3T5: eduPERT team

 

 

 

 

From: [mailto:] On Behalf Of Decker, Ryan C.
Sent: Freitag, 6. Oktober 2017 21:16
To: perfsonar-user
Subject: [perfsonar-user] graphs missing after upgrade to CentOS 7

 

Hey Guys,

I just finished upgrading from Centos 6.9 to 7.4 and i seem to be having trouble with the web interface graphs. I did a backup with ps-migrate-backup.sh and then restored on the new install and used the --data flag.

I immediately had an issue where all of the measurement_archive sections in /etc/perfsonar/meshconfig-agent-tasks.conf were duplicated. Which caused esmond to complain about invalid tokens.

 

That i believe i fixed with: /usr/lib/perfsonar/scripts/system_environment/configure_esmond --force after removing all of the measurement_archive sections.

 

What seems weird to me is that i can see results using the pscheduler command:

pscheduler result --diags --archivings https://66.194.180.9/pscheduler/tasks/2fc0e480-be9c-4652-8af4-abaf56b4fae8/runs/954825fb-cc85-4cf5-ad17-d268b479d4c9

 

Is there something else i need to do?

--

Ryan C. Decker '08
Principle Network & Systems Eng.
Siena College ITS
515 Loudon Road
Loudonville, NY 12211


CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you received this e-mail and are not the intended recipient, please inform the sender by e-mail reply and destroy all copies of the original message.

--

Ryan C. Decker '08
Principle Network & Systems Eng.
Siena College ITS
515 Loudon Road
Loudonville, NY 12211


CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you received this e-mail and are not the intended recipient, please inform the sender by e-mail reply and destroy all copies of the original message.

--

Ryan C. Decker '08
Principle Network & Systems Eng.
Siena College ITS
515 Loudon Road
Loudonville, NY 12211


CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you received this e-mail and are not the intended recipient, please inform the sender by e-mail reply and destroy all copies of the original message.