perfSONAR User Q&A and Other Discussion

[Hyojoon Kim <>]

Hi Shawn, 

Thank you! Your points were spot-on. 

The specific RPM was installed in the first place, but scripts sits in “/usr/lib64/...” instead of “/usr/lib/… ”. The reference in maddash.yaml was incorrect, pointing to “/usr/lib/”. 

Restarting the cron job manually did not fix the incorrect reference in maddash.yaml, so I decided to open "/etc/perfsonar/meshconfig-guiagent.conf” and uncomment the lines related to the check commands (in <maddash_options> block), making it point to “/usr/lib64/”. The default seems to be “/usr/lib64/”, but maybe leaving this as default does not force a modification in maddash.yaml. Making it explicit (by un-commmeting) seems to force and modify “maddash.yaml” to point to the right location. 

Now, blocks in our MaDDash are all becoming “green” again :-)  

Thanks again!  

Cheers, 

Joon 

On Mar 5, 2016, at 11:37 AM, Shawn McKee <> wrote:

Some further info.

The check_owdelay.pl command should be referenced in your /etc/maddash/maddash-server/maddash.yaml file.   This should get updated when your crontask 
in /etc/cron.d/perfsonar-meshconfig-guiagent runs.  Check this config file as well /etc/perfsonar/meshconfig-guiagent.conf

If this ran before your update on the maddash server that may be the problem.  Maybe kick-off that cron job right now and see if it fixes things?

Shawn

On Sat, Mar 5, 2016 at 11:30 AM, Shawn McKee <> wrote:
Hi Joon,

Do you have this RPM installed?

nagios-plugins-perfsonar-3.5.1-1.x86_64

If so you may have the path to the check in usr/lib64...

[root@maddash ~]# rpm -ql nagios-plugins-perfsonar-3.5.1-1.x86_64 | grep check_owdelay
/usr/lib64/nagios/plugins/check_owdelay.pl

That doesn't explain why MaDDash isn't configured to find it in the right location though.

Shawn

On Sat, Mar 5, 2016 at 11:00 AM, Hyojoon Kim <> wrote:
Thank you, Andy!  I see results coming in now, starting today at 2am :-) 

But now MaDDash seems to be seeing some errors. The check seems to fail, and I “/var/log/maddash/maddash-server.log” has messages like: 

ERROR 2016-03-05 10:49:16,660 Error running nagios check: Cannot run program "/usr/lib/nagios/plugins/check_owdelay.pl": error=2, No such file or directory

I tried restarting maddash-server, but I still see the same messages coming up. I see that there is indeed *no* file named “/usr/lib/nagios/plugins/check_owdelay.pl”. In fact there is no "/usr/lib/nagios” directory.

MaDDash seems upgraded to 1.3.-1.

[perfsonar-ma-2 maddash]# yum list installed | grep maddash
maddash.noarch          1.3-1           @Internet2                              
maddash-server.noarch   1.3-1           @Internet2                              
maddash-webui.noarch    1.3-1           @Internet2 


Any help would be great! I’m glad that we are now collecting data correctly and this is just a MaDDash display issue!

Thanks,
Joon

On Mar 5, 2016, at 1:17 AM, Christopher J. Tengi <> wrote:

Andy,
    Thanks for fixing the problem so quickly. This is what I call excellent customer service. I'll try to check over the weekend to see how things go.

/Chris


On Mar 4, 2016, at 10:34 PM, Andrew Lake <> wrote:

Hi Joon, Chris, Dengfeng,

Thanks for digging into his and reporting it. Turns out it was a bug with the regular testing client. I was able to recreate locally and as Joon’s last email was hinting at, the client was sending a PUT when it should have been sending a POST when Auth headers were not set (i.e. you’re using IP authentication). For the benefit of others reading this thread, standard toolkits using a local MA communicate with API key authentication so were not affected by this bug. This generally only affects those who use a central MA AND have chosen to use IP authentication. 

I have just pushed a fix to the main yum repo, it should start showing up on mirrors shortly and auto-updates will grab it shortly thereafter. You’ll know a client has the fixed version when they are running 3.5.1.1 of the libperfsonar-regulartesting-perl and perfsonar-regulartesting packages…or you start seeing data again :) I apologize for the inconvenience this caused and thank you for promptly reporting it so we could get it fixed before it plagued others trying a similar setup. Please let us know if you see any further issues after the update.

Thanks, 
Andy 



On March 4, 2016 at 6:43:21 PM, Hyojoon Kim () wrote:

I see that the central MA has Django installed and running and all, and esmond version is v2.0.


I’m not sure if this helps, but a packet capture of a message from the central MA to the client has this:

*Method “PUT” not allowed.* 


Thanks, 
Joon 


<>

On Mar 4, 2016, at 5:17 PM, Christopher J. Tengi <> wrote:

All of the testing nodes *and* the MA are full toolkit installs, and they all updated overnight.

/Chris


On Mar 4, 2016, at 4:49 PM, Andrew Lake <> wrote:

Hi,

Have you upgraded the central MA to the latest esmond? Both the client and the central MA need to be updated. That is likely causing the errors, since one of the HTTP methods changed, it will lead to the 405 error you are seeing. It may likely be contributing to the 401 error you are seeing, let us know if that persists after the upgrade. 

Thanks,
Andy



On March 4, 2016 at 4:41:35 PM, Christopher J. Tengi () wrote:

In the early hours of the morning, all of our pS toolkit hosts upgraded themselves, including our Measurement Archive. The test nodes are happily writing th their local esmonds, but they are failing to write to the central MA. We are using IP-based authentication, and are seeing the following in the regulartesting.log file on the test nodes: 

========== 
2016/03/04 16:19:55 (21107) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
2016/03/04 16:19:55 (21108) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 


Even though we are not runinng any tests on the MA, we are seeing something similar (but not the same) in the regulartesting.log file on the MA: 

========== 
2016/03/04 16:21:14 (28244) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (401) 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 401 UNAUTHORIZED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 

Any suggestions as to where I should look next? 

Thanks, 
/Chris