perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

Another problem with 4.2 :-(

After a scheduled test has finished (according to "pscheduler schedule"), I tried to retrieve its results, and I got an exception:

brian@perf1:~$ pscheduler result https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723
2019-09-05T11:22:51Z on ns1.BBBB.com and [A:A:A:A::45] with iperf3:

throughput --duration PT20S --source ns1.BBBB.com --ip-version 6 --dest A:A:A:A::45
  --source-node ns1.BBBB.com --dest-node '[A:A:A:A::45]'

Traceback (most recent call last):
  File "/usr/lib/pscheduler/commands/result", line 399, in <module>
    print_run_url=False
  File "/usr/lib/pscheduler/commands/result", line 252, in dump_run
    succeeded = run_json['result-merged']['succeeded']
TypeError: 'NoneType' object has no attribute '__getitem__'

Running 'curl' on that URL gives what seems like a reasonable output:

brian@perf1:~$ curl -k https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723
{"result-href": "https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723/result", "priority": 0, "added": "2019-09-04T14:27:03Z", "start-time": "2019-09-05T11:22:51Z", "participant-data": {"server_port": 5201}, ... snip rest ...

But I note that it explicitly contains:

    "result-merged": null,

which is what is causing the results script to barf:

    if run_ran:
        try:
            succeeded = run_json['result-merged']['succeeded']
            if succeeded is None:
                raise KeyError()
        except KeyError as ex:
            pscheduler.fail("Data missing from run result")

One immediate problem here is that the exception is not being caught properly: since run_json['result_merged'] exists but is null.  It needs to test for this, or catch TypeError.

However in any case, I know result run was successful.  Here are its top-level elements:

brian@perf1:~$ curl -sk https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723 | python3 -mjson.tool | grep "^    [^ ]" | awk '{ print $1,$2; }'
"result-href": "https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723/result",
"priority": 0,
"added": "2019-09-04T14:27:03Z",
"start-time": "2019-09-05T11:22:51Z",
"participant-data": {
},
"result-merged": null,
"participant-data-full": [
],
"state": "finished",
"state-display": "Finished",
"result-full": null,
"participants": [
],
"errors": null,
"href": "https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723",
"task-href": "https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4",
"duration": "PT23S",
"participant": 1,
"end-time": "2019-09-05T11:23:14Z",
"limit-diags": "Hints:\n
"result": {
}

>>> print(j.keys())
dict_keys(['result-href', 'priority', 'added', 'start-time', 'participant-data', 'result-merged', 'participant-data-full', 'state', 'state-display', 'result-full', 'participants', 'errors', 'href', 'task-href', 'duration', 'participant', 'end-time', 'limit-diags', 'result'])
>>> print(j["result"].keys())
dict_keys(['diags', 'intervals', 'succeeded', 'summary'])

Note that there is ["result"]["succeeded"], but not ["result-merged"]["succeeded"].

    "result": {
        ...
        "succeeded": true,
        "summary": {
            "streams": [
                {
                    "throughput-bytes": 0,
                    "start": 0,
                    "end": 20.031353,
                    "throughput-bits": 0,
                    "stream-id": 5
                }
            ],
            "summary": {
                "throughput-bytes": 0,
                "start": 0,
                "end": 20.031353,
                "throughput-bits": 0
            }
        }

And a lot of interval data:

        "intervals": [
            {
                "streams": [
                    {
                        "end": 1.000038,
                        "stream-id": 5,
                        "omitted": false,
                        "start": 0,
                        "throughput-bits": 204410850.71253952,
                        "throughput-bytes": 25552328
                    }
                ],
                "summary": {
                    "throughput-bytes": 25552328,
                    "start": 0,
                    "end": 1.000038,
                    "throughput-bits": 204410850.71253952,
                    "omitted": false
                }
            },

Oddly, result.diags consists of some lines of text followed by some more JSON (all wrapped in a string)

>>> print(j["result"]["diags"])
Unable to use NUMA for eth0.  Disabling it.
/usr/bin/iperf3 -s -1 --json -p 5201 -B A:A:A:A::45 -6

{
    "start":    {
        "connected":    [{
                "socket":    5,
                "local_host":    "A:A:A:A::45",
                "local_port":    5201,
                "remote_host":    "B:B:B:B:feff:ff:fe00:f16",
                "remote_port":    42917
            }],
        "version":    "iperf 3.7",
        "system_info":    "Linux perf1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64",
... snip rest ...
... the top level members are "start" and "intervals" only

I tried changing the code to succeeded = run_json['result']['succeeded'].  After this I got the following returned from pscheduler result:

Problem fetching results: Resource Not found.

So I changed the code again, to see what resource it was trying to access:

    except Exception as ex:
        print_message("Problem fetching results from %s: %s" % (url, str(ex)), exit_on_error)
        return

Now I get:

Problem fetching results from https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723/result: Resource Not found.

OK, so trying various URLs with curl:

brian@perf1:~$ curl -sk https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs
["https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/715d4882-c796-4393-9561-8cc5bf958b63", "https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/540cc1c1-170c-4939-a361-1725f60fe175", "https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723"]

brian@perf1:~$ curl -sk https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723
==> this works, produces a load of JSON data (as already shown)

brian@perf1:~$ curl -sk https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>

brian@perf1:~$ curl -sk https://localhost/pscheduler/tasks/447cbdae-d922-4190-9032-9232536169c4/runs/5687f64a-0dd0-49d9-9554-05c48e806723/result
Resource Not found.

Since I've no idea what to expect here, I can't see if these are right or not; but it seems to me that there's nothing available at the URL with the /result suffix.

I note that it's only following the "result-href" from the task JSON.

Any ideas from someone who knows how these bits are supposed to hang together?

Thanks,

Brian Candler.