perfSONAR User Q&A and Other Discussion

["Andrew Lake" <>]

Hi Phillippe, 

There have also been significant changes to the WLCG mesh that happened about the same time as the RC2 RPMs so you might just be running more tests. I debugged the same problem last week on a few of the other 4GB hosts that were similarly configured  (e.g. psum01), this is exactly what occurred to them. 

We were able to free up some more resources by removing a log scraper cron whose results are sparsely used but eats up a lot of resources. You can do the same with the following:

rm /etc/cron.hourly/logscraper.cron

That seemed to allow it to run more tests. We will be removing this before the final and documenting how to reinstall it for anyone that misses it and had enough overhead to run it. Depending on how far gone your host is you may also want to run the following (the equivalent commands will also be run as part of the regular restart overnight):

/etc/init.d/regular_testing stop

rm -rf /var/lib/perfsonar/regular_testing/*

/etc/init.d/regular_testing start

Hope that helps,

Andy

On Fri, Sep 4, 2015 at 3:48 PM, Philippe Laurens <> wrote:

Hi,

> - To update an RC1 host you should not need to take any action. Auto-updates should be on and you should already have the release. You may run “yum update” if you disabled auto-updates for any reason.

If the RC1 to RC2 transition happens via auto-update,
is a reboot recommended afterwards?

The MSU testbed psmsu05 and psmsu06.aglt2.org were auto-updated
on Sept 1st. Two days later we noticed extensive memory usage
on our OMD monitoring :

> psmsu05 (testbed for latency)
> WARN Memory used
> 6.79 GB used (3.51 RAM + 3.22 SWAP + 0.05 Pagetables, this is 181.6% of 3.74 RAM (3.94 total SWAP)), warning at 150.0% used, 0.0 mapped, 7.2 committed, 0.0 shared
>
> psmsu06 (testbed for bandwidth)
> WARN Memory used
> 5.89 GB used (3.57 RAM + 2.28 SWAP + 0.04 Pagetables, this is 157.5% of 3.74 RAM (3.94 total SWAP)), warning at 150.0% used, 0.0 mapped, 9.3 committed, 0.0 shared

Rebooting helped that crisis but there still seems
to be something about RC2 and memory usage,
cf. attached OMD graphs.

Thanks,

Philippe


On 02-Sep-2015 11:04 AM, Andrew Lake wrote:
> perfSONAR community,
>
> The second release candidate of the next major revision of perfSONAR is
> now available for beta testing. *Do NOT install this release on your
> production hosts as it is not a final release and is intended for
> testing purposes only.* We expect a final release in about a month, and
> this will hopefully by the last release candidate before then (subject
> to change based on your testing results of course).
>
> For a complete list of features and changes in this release candidate,
> see http://www.perfsonar.net/release-notes/version-3-5-rc2
> <http://www.perfsonar.net/release-notes/version-3-5-rc2>. A few
> highlights of changes from RC1 are:
>
> - *Further web interface enhancements: *The new web interface is now the
> main interface found at the /toolkit URL of your web browser. The
> administrative interface has been converted to the new layout and
> contains a number of UI enhancements.
> - *New Puppet Module: *A new Puppet module has been added to support
> larger deployments of hosts. Documentation for this module starts here:
> http://docs.perfsonar.net/3.5rc2/multi_puppet_overview.html
> <http://docs.perfsonar.net/3.5rc2/multi_puppet_overview.html>
> *- Bug fixes, bug fixes, and more bug fixes:* Numerous bug fixes related
> to multi-homing, lookup service registration, central management and
> more have been added based on feedback from the first release candidate.
> If you reported one, please verify it’s fixed and let us know if you
> find any new ones!
>
> Those are just some of the changes, see the release notes (again at
> http://www.perfsonar.net/release-notes/version-3-5-rc2
> <http://www.perfsonar.net/release-notes/version-3-5-rc2>) for further
> information. The full release notes also contain a link to the previous
> RC1 release notes that highlight changes since 3.4.2 if you would like
> to review those as well.
>
> To install the current release candidate do one of the following:
> - You may install a completely new Toolkit node by downloading and
> installing one of the ISO images here:
> http://downloads.perfsonar.net/toolkit/release_candidates/3.5rc2/
> <http://downloads.perfsonar.net/toolkit/release_candidates/3.5rc2/>:
> - To update an existing 3.4.X system run the following:
> yum install Internet2-repo-staging
> yum clean all
> yum update
> - To update an RC1 host you should not need to take any action.
> Auto-updates should be on and you should already have the release. You
> may run “yum update” if you disabled auto-updates for any reason.
>
> In order to help direct your testing, you may find a checklist for
> Toolkit install here
> https://github.com/perfsonar/project/wiki/Toolkit-Testing-Checklist
> <https://github.com/perfsonar/project/wiki/Toolkit-Testing-Checklist>.
> For documentation on this release candidate
> see http://docs.perfsonar.net/3.5rc2/.
>
> As always if you have any questions please contact our user mailing
> list. We appreciate you help in testing this release and look forward to
> your feedback.
>
> Thank you,
> The perfSONAR Team
>
>
<20150904_psmsu05_memory_usage.png><20150904_psmsu06_memory_usage.png>

<20150904_psmsu05_memory_usage.png><20150904_psmsu06_memory_usage.png>