perfSONAR User Q&A and Other Discussion

["Fligor, Debbie" <>]

Hi Mark,
sorry this has taken so long.

> On Apr 5, 2019, at 13:18, Mark Feit <> wrote:
> 
> Fligor, Debbie writes:
>  
> […]

> Here is one task:
> 
> on host res-dtn in /var/log/perfsonar/psconfig-pscheduler-agent-tasks.log
> 
> 2019/04/03 13:36:54 INFO guid=744C238C-563F-11E9-8763-FF999454770F 
> task_name=res-perfsonar_0_0 
> config_file=/etc/perfsonar/psconfig/pscheduler.d/toolkit-webui.json 
> config_src=include 
> task={"archives":[{"data":{"measurement-agent":"72.36.81.132","url":"https://72.36.81.132/esmond/perfsonar/archive/","_auth-token”:******"},"archiver":"esmond"}],"test":{"spec":{"source":"res-perfsonar.techservices.illinois.edu","dest":"72.36.81.132","dest-node":"72.36.81.132","duration":"PT90S","schema":1,"source-node":"res-perfsonar.techservices.illinois.edu","ip-version":4},"type":"throughput"},"reference":{"psconfig":{"created-by":{"user-agent":"psconfig-pscheduler-agent","uuid":"ECA79EF0-F98E-11E8-8308-CE3177A9DBB1"}}},"tools":["bwctliperf3","iperf3","bwctliperf2","iperf"],"schema":1,"schedule":{"until":"2019-04-04T18:36:54Z","sliprand":true,"repeat":"PT600S","slip":"PT600S"}}
> 
> on host res-perfsonar, in /var/log/pscheduler/pscheduler.log
> 
> Apr  3 13:37:01 res-perfsonar journal: pscheduler-api INFO     Denied 
> 72.36.81.132 POST 
> https://res-perfsonar.techservices.illinois.edu/pscheduler/tasks/f5bc3a93-876f-45e5-9c4c-f734e9f6b278
> Apr  3 13:37:02 res-perfsonar journal: pscheduler-api INFO     Denied 
> 72.36.81.132 POST 
> https://res-perfsonar.techservices.illinois.edu/pscheduler/tasks
> 
> 
>> One thing I just noticed is that the API is a bit inconsistent in how it 
>> logs errors.  Many of them include a message that’s logged during 
>> debugging but not in the lines you see above, which isn’t particularly 
>> helpful.  I’ve opened a ticket to get that improved.  In the meantime, you 
>> should be able to find a companion entry with the same GUID in 
>> psconfig-pscheduler-agent-transactions.log that will list the error it got 
>> back from the server.  Let me know what you find there.
>> 

thanks for opening the ticket on the API messages. Here’s what I found for 
that guid in the res-dtn host’s psconfig-pscheduler-agent-transactions.log 
file that didn’t include “Application PASSES” and was talking about 
res-perfsonar:


psconfig-pscheduler-agent-transactions.log.2:2019/04/03 13:36:54 INFO 
guid=744C238C-563F-11E9-8763-FF999454770F action=list 
url=https://res-perfsonar.techservices.illinois.edu/pscheduler msg=Getting 
task list from https://res-perfsonar.techservices.illinois.edu/pscheduler

psconfig-pscheduler-agent-transactions.log.2:2019/04/03 13:36:59 INFO 
guid=744C238C-563F-11E9-8763-FF999454770F action=create 
checksum=YQ9t+s3Fl1tt+vQBNDCGvQ test_type=throughput 
lead_url=https://72.36.81.132/pscheduler 
task={"archives":[{"data":{"measurement-agent":"72.36.81.132","url":"https://72.36.81.132/esmond/perfsonar/archive/","_auth-token":"6a8c215b79306f8e912969cdc1f8e79daee173a7"},"archiver":"esmond"}],"test":{"spec":{"source":"72.36.81.132","dest":"res-perfsonar.techservices.illinois.edu","dest-node":"res-perfsonar.techservices.illinois.edu","duration":"PT90S","schema":1,"source-node":"72.36.81.132","ip-version":4},"type":"throughput"},"reference":{"psconfig":{"created-by":{"user-agent":"psconfig-pscheduler-agent","uuid":"ECA79EF0-F98E-11E8-8308-CE3177A9DBB1"}}},"tools":["bwctliperf3","iperf3","bwctliperf2","iperf"],"schema":1,"schedule":{"until":"2019-04-04T18:36:54Z","sliprand":true,"repeat":"PT600S","slip":"PT600S”}}

psconfig-pscheduler-agent-transactions.log.2:2019/04/03 13:37:02 ERROR 
guid=744C238C-563F-11E9-8763-FF999454770F action=create msg=Problem adding 
test throughput(72.36.81.132->res-perfsonar.techservices.illinois.edu), 
continuing with rest of config: HTTP/1.1 500 INTERNAL SERVER ERROR

psconfig-pscheduler-agent-transactions.log.2:2019/04/03 13:37:02 INFO 
guid=744C238C-563F-11E9-8763-FF999454770F action=create 
checksum=p4DNoKkCTVqMfOuH2LNKPg test_type=throughput 
lead_url=https://res-perfsonar.techservices.illinois.edu/pscheduler 
task={"archives":[{"data":{"measurement-agent":"72.36.81.132","url":"https://72.36.81.132/esmond/perfsonar/archive/","_auth-token":"6a8c215b79306f8e912969cdc1f8e79daee173a7"},"archiver":"esmond"}],"test":{"spec":{"source":"res-perfsonar.techservices.illinois.edu","dest":"72.36.81.132","dest-node":"72.36.81.132","duration":"PT90S","schema":1,"source-node":"res-perfsonar.techservices.illinois.edu","ip-version":4},"type":"throughput"},"reference":{"psconfig":{"created-by":{"user-agent":"psconfig-pscheduler-agent","uuid":"ECA79EF0-F98E-11E8-8308-CE3177A9DBB1"}}},"tools":["bwctliperf3","iperf3","bwctliperf2","iperf"],"schema":1,"schedule":{"until":"2019-04-04T18:36:54Z","sliprand":true,"repeat":"PT600S","slip":"PT600S”}}

psconfig-pscheduler-agent-transactions.log.2:2019/04/03 13:37:02 ERROR 
guid=744C238C-563F-11E9-8763-FF999454770F action=create msg=Problem adding 
test throughput(res-perfsonar.techservices.illinois.edu->72.36.81.132), 
continuing with rest of config: HTTP/1.1 403 FORBIDDEN


>> I ran a few permutations of the tests you’re doing manually (as you did) 
>> and found that they worked just fine.  Because of that, I suspect this 
>> might be a case of pScheduler being told to do something it doesn’t like 
>> by pSConfig, either because of a bug there or a GIGO problem.  Either way, 
>> we’ll get to the bottom of it.
>> 


My current thinking is that I’ve messed around too much, I should re-install 
them both, so they’re in a theoretically known good state, and start from 
there. It seems not worth a lot of time to debug a configuration that there 
is some chance I’ve messed up part of when I was randomly poking at things 
trying to get them to work.  then if they still don’t work, debug from there. 

Thoughts?

-- 
-debbie
Debbie Fligor, n9dn       Lead Network Engineer @ Univ. of Il
email: