perfSONAR User Q&A and Other Discussion

["Fligor, Debbie" <>]

Hi Ivan,
I was mostly asking about if I was attempting to turn on debugging correctly 
or if I’d hit a bug. Thanks for confirmation that it should have worked the 
way I did it, and I’ll try your workaround suggestion.

If you (or someone else) is up for helping with my other problem, even though 
I haven’t finished exhaustive debugging, that would be great.  I’ve added 
some details here, but am happy to start a separate thread if that’s better.

my pscheduler problem is specifically between my two new hosts, which are on 
the same L2 network, and seem to be trying to talk to each other over IPv6, 
even for IPv4 tests. I tried turning off all the IPv6 tests to see if that 
helped, but it didn’t.  

 Scheduled tests to two other hosts not on the same network work fine from 
both my new hosts, and running the test by hand with pscheduler task works 
fine (and seems to be talking over IPv6 as well) between my two hosts.  Only 
the scheduled test is failing.  I had it working at one point, early in the 
deployment of these boxes, but it stopped working while I was troubleshooting 
other issues that turned out to be dirty fiber. I made a lot of changes 
during that troubleshooting, and it working came and went, as did my ability 
to successfully configure tests via the gui (I have another thread on that 
where I posted the workaround I finally found for that bug). 

I’ve mostly been looking through various log files trying to figure out 
exactly what is failing, so I can narrow it down to what isn’t working right 
and don’t have many insights yet


One thing I tried, was removing this line from bogons/exclude section of 
/etc/pscheduler/limits.conf:

                    "fe80::/10"

since I thought perhaps being on the same L2 network they might be trying to 
use local addresses, but it didn’t help.

After that I turned off IPv6 for the “res-dtn” host, and the log entries say 
IPv4 addresses, and I get the same denied error. So I don’t think it’s IP 
version related.

If there’s a known reason two hosts on the same subnet can’t run throughput 
tests, that might be what I’m hitting. Otherwise any thoughts you have are 
welcome.  Below are things I hope are useful/what you were asking for.

-debbie


Here is one task:

on host res-dtn in /var/log/perfsonar/psconfig-pscheduler-agent-tasks.log

2019/04/03 13:36:54 INFO guid=744C238C-563F-11E9-8763-FF999454770F 
task_name=res-perfsonar_0_0 
config_file=/etc/perfsonar/psconfig/pscheduler.d/toolkit-webui.json 
config_src=include 
task={"archives":[{"data":{"measurement-agent":"72.36.81.132","url":"https://72.36.81.132/esmond/perfsonar/archive/","_auth-token”:******"},"archiver":"esmond"}],"test":{"spec":{"source":"res-perfsonar.techservices.illinois.edu","dest":"72.36.81.132","dest-node":"72.36.81.132","duration":"PT90S","schema":1,"source-node":"res-perfsonar.techservices.illinois.edu","ip-version":4},"type":"throughput"},"reference":{"psconfig":{"created-by":{"user-agent":"psconfig-pscheduler-agent","uuid":"ECA79EF0-F98E-11E8-8308-CE3177A9DBB1"}}},"tools":["bwctliperf3","iperf3","bwctliperf2","iperf"],"schema":1,"schedule":{"until":"2019-04-04T18:36:54Z","sliprand":true,"repeat":"PT600S","slip":"PT600S"}}

on host res-perfsonar, in /var/log/pscheduler/pscheduler.log

Apr  3 13:37:01 res-perfsonar journal: pscheduler-api INFO     Denied 
72.36.81.132 POST 
https://res-perfsonar.techservices.illinois.edu/pscheduler/tasks/f5bc3a93-876f-45e5-9c4c-f734e9f6b278
Apr  3 13:37:02 res-perfsonar journal: pscheduler-api INFO     Denied 
72.36.81.132 POST 
https://res-perfsonar.techservices.illinois.edu/pscheduler/tasks


Here’s a “worked fine” by-hand test on the same two hosts from a previous 
test, when IPv6 was still on:


pscheduler task --debug throughput --dest 
res-perfsonar.techservices.illinois.edu --source 
res-dtn.techservices.illinois.edu --ip-version 4

2019-03-19T16:12:39 Debug started
2019-03-19T16:12:39 Assistance is from localhost
2019-03-19T16:12:39 Forcing default slip of PT5M
2019-03-19T16:12:39 Converting to spec via 
https://localhost/pscheduler/tests/throughput/spec
Submitting task...
2019-03-19T16:12:39 Fetching participant list
2019-03-19T16:12:39 Spec is: {"dest": 
"res-perfsonar.techservices.illinois.edu", "source": 
"res-dtn.techservices.illinois.edu", "ip-version": 4, "schema": 1}
2019-03-19T16:12:39 Params are: {'spec': '{"dest": 
"res-perfsonar.techservices.illinois.edu", "source": 
"res-dtn.techservices.illinois.edu", "ip-version": 4, "schema": 1}'}
2019-03-19T16:12:40 Got participants: {u'participants': 
[u'res-dtn.techservices.illinois.edu', 
u'res-perfsonar.techservices.illinois.edu']}
2019-03-19T16:12:40 Lead is res-dtn.techservices.illinois.edu
2019-03-19T16:12:40 Pinging 
https://res-dtn.techservices.illinois.edu/pscheduler/
2019-03-19T16:12:40 res-dtn.techservices.illinois.edu is up
2019-03-19T16:12:40 Posting task to 
https://res-dtn.techservices.illinois.edu/pscheduler/tasks
2019-03-19T16:12:40 Data is {"test": {"type": "throughput", "spec": {"dest": 
"res-perfsonar.techservices.illinois.edu", "source": 
"res-dtn.techservices.illinois.edu", "ip-version": 4, "schema": 1}}, 
"schema": 1, "schedule": {"slip": "PT5M"}}
Task URL:
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df
2019-03-19T16:12:42 Posted 
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df
2019-03-19T16:12:42 Submission diagnostics:
2019-03-19T16:12:42   Hints:
2019-03-19T16:12:42     requester: 2620:0:e01:8001::20
2019-03-19T16:12:42     server: 2620:0:e01:8001::20
2019-03-19T16:12:42   Identified as everybody, local-interfaces
2019-03-19T16:12:42   Classified as default, friendlies
2019-03-19T16:12:42   Application: Hosts we trust to do everything
2019-03-19T16:12:42     Group 1: Limit 'always' passed
2019-03-19T16:12:42     Group 1: Want all, 1/1 passed, 0/1 failed: PASS
2019-03-19T16:12:42     Application PASSES
2019-03-19T16:12:42   Application: Defaults applied to non-friendly hosts
2019-03-19T16:12:42     Group 1: Limit 'innocuous-tests' failed: Passed but 
inverted
2019-03-19T16:12:42     Group 1: Limit 'throughput-default-time' passed
2019-03-19T16:12:42     Group 1: Limit 'idleex-default' failed: Test is not 
'idleex'
2019-03-19T16:12:42     Group 1: Want any, 1/3 passed, 2/3 failed: PASS
2019-03-19T16:12:42     Application PASSES
2019-03-19T16:12:42   Proposal meets limits
Running with tool 'iperf3'
Fetching first run...
2019-03-19T16:12:42 Fetching 
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df/runs/first
2019-03-19T16:12:43 Handing off: pscheduler watch --first --format text/plain 
--debug 
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df
2019-03-19T16:12:43 Debug started
2019-03-19T16:12:43 Fetching 
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df
2019-03-19T16:12:43 Fetching next run from 
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df/runs/first

Next scheduled run:
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df/runs/b65ede71-ee0e-495d-9c66-c935dc292f95
Starts 2019-03-19T21:12:52Z (~8 seconds)
Ends   2019-03-19T21:13:11Z (~18 seconds)
Waiting for result...

* Stream ID 5
Interval       Throughput     Retransmits    Current Window 
0.0 - 1.0      33.21 Gbps     0              4.21 MBytes    
1.0 - 2.0      32.27 Gbps     53             1.18 MBytes    
2.0 - 3.0      23.07 Gbps     166            1.17 MBytes    
3.0 - 4.0      28.84 Gbps     192            1.51 MBytes    
4.0 - 5.0      27.47 Gbps     473            832.16 KBytes  
5.0 - 6.0      22.56 Gbps     567            1.20 MBytes    
6.0 - 7.0      33.42 Gbps     0              1.35 MBytes    
7.0 - 8.0      33.39 Gbps     0              1.35 MBytes    
8.0 - 9.0      33.38 Gbps     0              1.46 MBytes    
9.0 - 10.0     37.10 Gbps     89             1.30 MBytes    

Summary
Interval       Throughput     Retransmits    
0.0 - 10.0     30.47 Gbps     1540
2019-03-19T16:13:12 Fetching next run from 
https://res-dtn.techservices.illinois.edu/pscheduler/tasks/ab3bcb60-f11e-49a8-98d0-d3c01f3974df/runs/next

No further runs scheduled.









> On Apr 3, 2019, at 6:12, Garnizov, Ivan <> wrote:
> 
> Dear Debbie,
>  
> I see your inquiry as twofold:
> -          You are having an issue with pscheduler test coordination
> -          There is a bug in pscheduler on activating debugging mode for api
>  
> Your report about the bug will be reported to the pS Development team.
> On the issue with pscheduler tests coordination please share your 
> pscheduler request and pscheduler output.
> An additional assistance to us would be, if you execute the pscheduler 
> request with --debug options.
> (Example)
> pscheduler task --debug <<your test request here>>
>  
> As a workaround for the bug, you found, I would suggest to just run the 
> commands below with sudo:
> -          Activate debug: pscheduler debug on
> -          Deactivate debug: pscheduler debug off
>  
> Regards,
> Ivan Garnizov
>  
> GEANT WP6T3: pS development team
> GEANT WP7T1: pS deployments GN Operations
> GEANT WP9T2: Software governance in GEANT
>  
> From:  
> [] On Behalf Of Fligor, Debbie
> Sent: Tuesday, April 2, 2019 11:05 PM
> To: 
> Cc: Fligor, Debbie <>
> Subject: [perfsonar-user] debugging pscheduler
>  
> I’ve been trying to work out why my two new perfsonar boxes can’t run 
> througput tests with each other. My main clue is this in the logfile (both 
> systems show the same error):
> 
> Apr  2 15:22:15 res-dtn journal: pscheduler-api INFO     Denied 
> 2620:0:e01:8001::10 POST 
> https://res-dtn.techservices.illinois.edu/pscheduler/tasks/b20f8a9a-502b-4070-8625-daf960bebc1e
> Apr  2 15:22:16 res-dtn journal: pscheduler-api INFO     Denied 
> 2620:0:e01:8001::10 POST 
> https://res-dtn.techservices.illinois.edu/pscheduler/tasks
> 
> 
> 
> So I turned on debugging for the api in one of them, hoping to get more 
> details about what was failing, with this command:
> 
> 
>  pscheduler debug on api
> 
> which I found on this page:
> 
> http://docs.perfsonar.net/troubleshooting_overview.html
> 
> 
> and got this in the log file:
> 
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR    Exception: 
> IOError: [Errno 2] No such file or directory: 
> '/run/pscheduler-server/pscheduler-api-debug'
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR    Traceback (most 
> recent call last):
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in 
> full_dispatch_request
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR        rv = 
> self.dispatch_request()
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR      File 
> "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in 
> dispatch_request
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR        return 
> self.view_functions[rule.endpoint](**req.view_args)
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR      File 
> "./pschedulerapiserver/debug.py", line 77, in debug
> Apr  2 15:58:14 res-dtn journal: pscheduler-api ERROR    IOError: [Errno 2] 
> No such file or directory: '/run/pscheduler-server/pscheduler-api-debug'
> 
> Should I be able to turn on API debugging with that command, or did I need 
> other settings?
> 
> 
> -- 
> -debbie
> Debbie Fligor, n9dn       Lead Network Engineer @ Univ. of Il
> email:          
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: 
> https://lists.internet2.edu/sympa/signoff/perfsonar-user

-- 
-debbie
Debbie Fligor, n9dn       Lead Network Engineer @ Univ. of Il
email: