Skip to Content.
Sympa Menu

perfsonar-user - Re: [perfsonar-user] problems writing to central MA using IP-based authentication after perfSONAR 3.5.1 automatic upgrade

Subject: perfSONAR User Q&A and Other Discussion

List archive

Re: [perfsonar-user] problems writing to central MA using IP-based authentication after perfSONAR 3.5.1 automatic upgrade


Chronological Thread 
  • From: Hyojoon Kim <>
  • To: "Christopher J. Tengi" <>
  • Cc: Andrew Lake <>, perfsonar-user <>
  • Subject: Re: [perfsonar-user] problems writing to central MA using IP-based authentication after perfSONAR 3.5.1 automatic upgrade
  • Date: Sat, 5 Mar 2016 16:00:56 +0000
  • Accept-language: en-US
  • Importance: high

Thank you, Andy!  I see results coming in now, starting today at 2am :-) 

But now MaDDash seems to be seeing some errors. The check seems to fail, and I “/var/log/maddash/maddash-server.log” has messages like: 

ERROR 2016-03-05 10:49:16,660 Error running nagios check: Cannot run program "/usr/lib/nagios/plugins/check_owdelay.pl": error=2, No such file or directory

I tried restarting maddash-server, but I still see the same messages coming up. I see that there is indeed *no* file named “/usr/lib/nagios/plugins/check_owdelay.pl”. In fact there is no "/usr/lib/nagios” directory.

MaDDash seems upgraded to 1.3.-1.

[perfsonar-ma-2 maddash]# yum list installed | grep maddash
maddash.noarch          1.3-1           @Internet2                              
maddash-server.noarch   1.3-1           @Internet2                              
maddash-webui.noarch    1.3-1           @Internet2 


Any help would be great! I’m glad that we are now collecting data correctly and this is just a MaDDash display issue!

Thanks,
Joon

On Mar 5, 2016, at 1:17 AM, Christopher J. Tengi <> wrote:

Andy,
    Thanks for fixing the problem so quickly. This is what I call excellent customer service. I'll try to check over the weekend to see how things go.

/Chris


On Mar 4, 2016, at 10:34 PM, Andrew Lake <> wrote:

Hi Joon, Chris, Dengfeng,

Thanks for digging into his and reporting it. Turns out it was a bug with the regular testing client. I was able to recreate locally and as Joon’s last email was hinting at, the client was sending a PUT when it should have been sending a POST when Auth headers were not set (i.e. you’re using IP authentication). For the benefit of others reading this thread, standard toolkits using a local MA communicate with API key authentication so were not affected by this bug. This generally only affects those who use a central MA AND have chosen to use IP authentication. 

I have just pushed a fix to the main yum repo, it should start showing up on mirrors shortly and auto-updates will grab it shortly thereafter. You’ll know a client has the fixed version when they are running 3.5.1.1 of the libperfsonar-regulartesting-perl and perfsonar-regulartesting packages…or you start seeing data again :) I apologize for the inconvenience this caused and thank you for promptly reporting it so we could get it fixed before it plagued others trying a similar setup. Please let us know if you see any further issues after the update.

Thanks, 
Andy 



On March 4, 2016 at 6:43:21 PM, Hyojoon Kim () wrote:


I see that the central MA has Django installed and running and all, and esmond version is v2.0.


I’m not sure if this helps, but a packet capture of a message from the central MA to the client has this:

*Method “PUT” not allowed.* 


Thanks, 
Joon 


<>




On Mar 4, 2016, at 5:17 PM, Christopher J. Tengi <> wrote:

All of the testing nodes *and* the MA are full toolkit installs, and they all updated overnight.

/Chris


On Mar 4, 2016, at 4:49 PM, Andrew Lake <> wrote:

Hi,

Have you upgraded the central MA to the latest esmond? Both the client and the central MA need to be updated. That is likely causing the errors, since one of the HTTP methods changed, it will lead to the 405 error you are seeing. It may likely be contributing to the 401 error you are seeing, let us know if that persists after the upgrade. 

Thanks,
Andy



On March 4, 2016 at 4:41:35 PM, Christopher J. Tengi () wrote:

In the early hours of the morning, all of our pS toolkit hosts upgraded themselves, including our Measurement Archive. The test nodes are happily writing th their local esmonds, but they are failing to write to the central MA. We are using IP-based authentication, and are seeing the following in the regulartesting.log file on the test nodes: 

========== 
2016/03/04 16:19:55 (21107) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
2016/03/04 16:19:55 (21108) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 


Even though we are not runinng any tests on the MA, we are seeing something similar (but not the same) in the regulartesting.log file on the MA: 

========== 
2016/03/04 16:21:14 (28244) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (401) 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 401 UNAUTHORIZED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 

Any suggestions as to where I should look next? 

Thanks, 
/Chris 





Archive powered by MHonArc 2.6.16.

Top of Page