perfSONAR User Q&A and Other Discussion

["Liu, Dengfeng (NIH/CIT) [C]" <>]

Thank you Andrew.

Dengfeng

From: Andrew Lake

Sent: Friday, March 4, 2016 10:34 PM

To: Christopher J. Tengi; Hyojoon Kim

Cc: perfsonar-user

Subject: Re: [perfsonar-user] problems writing to central MA using IP-based authentication after perfSONAR 3.5.1 automatic upgrade

Hi Joon, Chris, Dengfeng,

Thanks for digging into his and reporting it. Turns out it was a bug with the regular testing client. I was able to recreate locally and as Joon’s last email was hinting at, the client was sending a PUT when it should have been sending a POST when Auth headers were not set (i.e. you’re using IP authentication). For the benefit of others reading this thread, standard toolkits using a local MA communicate with API key authentication so were not affected by this bug. This generally only affects those who use a central MA AND have chosen to use IP authentication. 

I have just pushed a fix to the main yum repo, it should start showing up on mirrors shortly and auto-updates will grab it shortly thereafter. You’ll know a client has the fixed version when they are running 3.5.1.1 of the libperfsonar-regulartesting-perl and perfsonar-regulartesting packages…or you start seeing data again :) I apologize for the inconvenience this caused and thank you for promptly reporting it so we could get it fixed before it plagued others trying a similar setup. Please let us know if you see any further issues after the update.

Thanks, 

Andy 

On March 4, 2016 at 6:43:21 PM, Hyojoon Kim () wrote:

I see that the central MA has Django installed and running and all, and esmond version is v2.0.

I’m not sure if this helps, but a packet capture of a message from the central MA to the client has this:

*Method “PUT” not allowed.* 

Thanks, 

Joon 

On Mar 4, 2016, at 5:17 PM, Christopher J. Tengi <> wrote:

All of the testing nodes *and* the MA are full toolkit installs, and they all updated overnight.

/Chris

On Mar 4, 2016, at 4:49 PM, Andrew Lake <> wrote:

Hi,

Have you upgraded the central MA to the latest esmond? Both the client and the central MA need to be updated. That is likely causing the errors, since one of the HTTP methods changed, it will lead to the 405 error you are seeing. It may likely be contributing to the 401 error you are seeing, let us know if that persists after the upgrade. 

Thanks,

Andy

On March 4, 2016 at 4:41:35 PM, Christopher J. Tengi () wrote:

In the early hours of the morning, all of our pS toolkit hosts upgraded themselves, including our Measurement Archive. The test nodes are happily writing th their local esmonds, but they are failing to write to the central MA. We are using IP-based authentication, and are seeing the following in the regulartesting.log file on the test nodes: 

========== 
2016/03/04 16:19:55 (21107) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
2016/03/04 16:19:55 (21108) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 


Even though we are not runinng any tests on the MA, we are seeing something similar (but not the same) in the regulartesting.log file on the MA: 

========== 
2016/03/04 16:21:14 (28244) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (401) 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 401 UNAUTHORIZED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 

Any suggestions as to where I should look next? 

Thanks, 
/Chris