perfSONAR User Q&A and Other Discussion

["Uhl, George D. (GSFC-423.0)[SGT INC]" <>]

Sorry.  I meant everything worked fine until the upgrade from 3.5 to 3.5-1.

From: <> on behalf of George Uhl <>
Date: Wednesday, March 9, 2016 at 1:28 PM
To: Daniel Doyle <>
Cc: Andrew Lake <>, perfsonar-user <>
Subject: Re: [perfsonar-user] problems writing to central MA using IP-based authentication after perfSONAR 3.5.1 automatic upgrade

Dan,

I’m running Ubuntu 12.04.5 and I’m updating daily using cron-apt.  The perfsonar-regulartesting package is version 3.5-1.  The libperfsonar-regulartesting-perl package is not installed. Maybe because the LivaX was built with the pS-testpoint bundle?    Everything worked fine until the upgrade to 3.5.1.1. 

Thanks,

George

From: Daniel Doyle <>
Date: Wednesday, March 9, 2016 at 1:10 PM
To: George Uhl <>
Cc: Andrew Lake <>, perfsonar-user <>
Subject: Re: [perfsonar-user] problems writing to central MA using IP-based authentication after perfSONAR 3.5.1 automatic upgrade

George,

I assume the LivaX hosts are running Debian. Are they updated to the latest 3.5.1.1 of both “perfsonar-regulartesting” and “libperfsonar-regulartesting-perl”? It looks like builds were generated around the same time Andy did the CentOS build fixes but might not have been marked as official releases.

Dan Doyle
GlobalNOC Software Developer
1-812-856-3892

On Mar 9, 2016, at 10:24 AM, Uhl, George D. (GSFC-423.0)[SGT INC] <> wrote:

Andy,

My Centos servers are able to write to my Central MA but I’m still having similar problems with my LivaX hosts.  

OS: Ubuntu 12.04.5 LTS

pS Bundle: perfsonar-testpoint v 3.5-1

I’ve enabled the daily cron-apt updates.   Regular testing log is showing the 405 error that others have reported.

2016/03/09 09:52:07 (30055) ERROR> EsmondBase.pm:65 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing data (405) 405 METHOD NOT ALLOWED

2016/03/09 09:52:07 (30055) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing data: 405 METHOD NOT ALLOWED

2016/03/09 09:52:07 (30055) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing data: 405 METHOD NOT ALLOWED at /usr/share/perl5/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122.

Did the changes to the yum repo make it to the Ubuntu/Debian side?  dpkg  says I’m running 3.5-1 of the perfsonar-regulartesting package.

Thanks,

George

From: <> on behalf of Andrew Lake <>
Date: Friday, March 4, 2016 at 10:33 PM
To: "Christopher J. Tengi" <>, Hyojoon Kim <>
Cc: perfsonar-user <>
Subject: Re: [perfsonar-user] problems writing to central MA using IP-based authentication after perfSONAR 3.5.1 automatic upgrade

Hi Joon, Chris, Dengfeng,

Thanks for digging into his and reporting it. Turns out it was a bug with the regular testing client. I was able to recreate locally and as Joon’s last email was hinting at, the client was sending a PUT when it should have been sending a POST when Auth headers were not set (i.e. you’re using IP authentication). For the benefit of others reading this thread, standard toolkits using a local MA communicate with API key authentication so were not affected by this bug. This generally only affects those who use a central MA AND have chosen to use IP authentication. 

I have just pushed a fix to the main yum repo, it should start showing up on mirrors shortly and auto-updates will grab it shortly thereafter. You’ll know a client has the fixed version when they are running 3.5.1.1 of the libperfsonar-regulartesting-perl and perfsonar-regulartesting packages…or you start seeing data again :) I apologize for the inconvenience this caused and thank you for promptly reporting it so we could get it fixed before it plagued others trying a similar setup. Please let us know if you see any further issues after the update.

Thanks, 

Andy 

On March 4, 2016 at 6:43:21 PM, Hyojoon Kim () wrote:

I see that the central MA has Django installed and running and all, and esmond version is v2.0.

I’m not sure if this helps, but a packet capture of a message from the central MA to the client has this:

*Method “PUT” not allowed.* 

Thanks, 

Joon 

On Mar 4, 2016, at 5:17 PM, Christopher J. Tengi <> wrote:

All of the testing nodes *and* the MA are full toolkit installs, and they all updated overnight.

/Chris

On Mar 4, 2016, at 4:49 PM, Andrew Lake <> wrote:

Hi,

Have you upgraded the central MA to the latest esmond? Both the client and the central MA need to be updated. That is likely causing the errors, since one of the HTTP methods changed, it will lead to the 405 error you are seeing. It may likely be contributing to the 401 error you are seeing, let us know if that persists after the upgrade. 

Thanks,

Andy

On March 4, 2016 at 4:41:35 PM, Christopher J. Tengi () wrote:

In the early hours of the morning, all of our pS toolkit hosts upgraded themselves, including our Measurement Archive. The test nodes are happily writing th their local esmonds, but they are failing to write to the central MA. We are using IP-based authentication, and are seeing the following in the regulartesting.log file on the test nodes: 

========== 
2016/03/04 16:19:55 (21107) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21107) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
2016/03/04 16:19:55 (21108) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (405) 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED 
2016/03/04 16:19:55 (21108) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 405 METHOD NOT ALLOWED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 


Even though we are not runinng any tests on the MA, we are seeing something similar (but not the same) in the regulartesting.log file on the MA: 

========== 
2016/03/04 16:21:14 (28244) ERROR> EsmondBase.pm:53 perfSONAR_PS::RegularTesting::MeasurementArchives::EsmondBase::__ANON__ - Error writing metadata (401) 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:209 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::handle_results - Problem storing results: Error writing metadata: 401 UNAUTHORIZED 
2016/03/04 16:21:14 (28244) ERROR> MeasurementArchiveChild.pm:125 perfSONAR_PS::RegularTesting::Master::MeasurementArchiveChild::__ANON__ - Problem handling test results: Problem storing results: Error writing metadata: 401 UNAUTHORIZED at /usr/lib/perfsonar/bin/../lib/perfSONAR_PS/RegularTesting/Master/MeasurementArchiveChild.pm line 122. 
========== 

Any suggestions as to where I should look next? 

Thanks, 
/Chris