perfSONAR User Q&A and Other Discussion

[Michael Johnson <>]

Hi Daniel,

It looks like something isn't archiving correctly, either due to something in 
esmond, or something in pscheduler's esmond archiver.

If you run something like this, you may get some insight into what is going on
curl -k 
https://129.22.159.119/pscheduler/tasks/621269ac-75dd-4120-9d41-42372141e988/runs/f0d5082b-3813-41b9-a9a6-cd083995abb0?pretty

Notice I added ?pretty to the end of the URL to make it more readable. You 
might have to do this on one of the other results; pscheduler only retains 
this state info for 48 hours, so you might need to do it on something more 
recent.

In particular, look for an 'archivings' section and see if there is anything 
in there. If there were errors archiving, it should contain that information. 
Errors in other sections of the JSON result might be helpful as well.

You could also try enabling pscheduler's archive debugging:

sudo pscheduler debug on archiver

then running more tests and looking at /var/log/pscheduler/pscheduler.log

Other thoughts -- starting out vague, since I can't access your pscheduler or 
Esmond Web APIs.

Is cassandra running?
Is selinux disabled?

I'd also be curious if there is data here (for instance):
https://perfsonar-mixon.its.cwru.edu/esmond/perfsonar/archive/07a18ac7ca9245dbbffccefb601cbc5a/packet-trace/base

Thanks,
Michael

On Fri, Jan 18, 2019 at 11:26:55AM -0500, Daniel Matthews wrote:
> I have a single box that is exhibiting this behavior. Its newly installed.
> The pscheduler logs show tests are running. In the esmond archive data I am
> seeing all "time-updated"; null. I have checked my other perfsonar
> instances and time-updated is not null. I'm guessing the page looks for
> time updated to display test results, but have no idea why my esmond
> archive doesn't have time-updated data. Been running for longer than 24
> hours so NTP should be stable. running pscheduer tasks by hand works fine.
> Any ideas?
>
> Cheers
> ========================
> Dan Matthews
> Manager, Network Engineering and Unified Communications
> Case Western Reserve University
> Office 216-368-8834
> Mobile 330-671-0002
> ========================
>
> *GET* /esmond/perfsonar/archive/
>
> *HTTP 200 OK**Allow:* GET, POST, HEAD, OPTIONS*Content-Type:*
> application/json*Vary:* Accept
> [
>    {
>        "url": 
> "https://perfsonar-mixon.its.cwru.edu/esmond/perfsonar/archive/07a18ac7ca9245dbbffccefb601cbc5a/";,
>        "metadata-key": "07a18ac7ca9245dbbffccefb601cbc5a",
>        "subject-type": "point-to-point",
>        "event-types": [
>            {
>                "summaries": [],
>                "time-updated": null,
>                "base-uri":
> "/esmond/perfsonar/archive/07a18ac7ca9245dbbffccefb601cbc5a/packet-trace/base",
>                "event-type": "packet-trace"
>            },
>            {
>                "summaries": [],
>                "time-updated": null,
>                "base-uri":
> "/esmond/perfsonar/archive/07a18ac7ca9245dbbffccefb601cbc5a/failures/base",
>                "event-type": "failures"
>            },
>            {
>                "summaries": [],
>                "time-updated": null,
>                "base-uri":
> "/esmond/perfsonar/archive/07a18ac7ca9245dbbffccefb601cbc5a/path-mtu/base",
>                "event-type": "path-mtu"
>            },
>            {
>                "summaries": [],
>                "time-updated": null,
>                "base-uri":
> "/esmond/perfsonar/archive/07a18ac7ca9245dbbffccefb601cbc5a/pscheduler-run-href/base",
>                "event-type": "pscheduler-run-href"
>            }
>        ],
>        "source": "129.22.159.119",
>        "destination": "192.5.110.70",
>        "measurement-agent": "129.22.159.119",
>        "tool-name": "pscheduler/traceroute",
>        "input-source": "129.22.159.119",
>        "input-destination": "perfsonar.sci.cwru.edu",
>        "pscheduler-reference-psconfig-created-by-user-agent":
> "psconfig-pscheduler-agent",
>        "pscheduler-reference-psconfig-created-by-uuid":
> "C0BA584E-1A83-11E9-A43F-DF1BBB8FFC4B",
>        "pscheduler-test-type": "trace",
>        "time-duration": "8.0",
>        "uri": "/esmond/perfsonar/archive/07a18ac7ca9245dbbffccefb601cbc5a/",
>        "metadata-count-total": 38,
>        "metadata-previous-page": null,
>        "metadata-next-page": null
>    },
>
> === pscheduler.log ===
>
> Jan 18 11:11:20 localhost journal: runner INFO     294: Running
> https://129.22.159.119/pscheduler/tasks/621269ac-75dd-4120-9d41-42372141e988/runs/f0d5082b-3813-41b9-a9a6-cd083995abb0
> Jan 18 11:11:20 localhost journal: runner INFO     294: With ping: rtt
> --dest www.stanford.edu --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:11:24 localhost journal: runner INFO     294: Run succeeded.
> Jan 18 11:11:45 localhost journal: runner INFO     146: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7fb6555e-e0b4-4b13-8c5a-7eb904ce08ac/runs/e5d95df9-32a5-4fdd-9706-6ac3a09654c8
> Jan 18 11:11:51 localhost journal: runner INFO     3830: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7676e147-2c5c-4daa-94b4-f9e2f8fb56e1/runs/2483327e-f270-49c5-a63f-14ccdcbedabd
> Jan 18 11:11:58 localhost journal: runner INFO     174: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/b8bdcb8d-9c8e-4ce3-ae0e-64af9c3e03b1/runs/da929d2e-5560-49aa-9510-0b7a5099b661
> Jan 18 11:11:58 localhost journal: runner INFO     159: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/e9872ff1-073e-444a-a00b-8dcee282d703/runs/24e200e8-6aac-4cdc-8d38-9d9ce9fc02ca
> Jan 18 11:11:59 localhost journal: runner INFO     3832: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/ba349cad-f10e-40b9-b8b3-f802a3fd9a60/runs/90f5536d-bf45-4d8f-81a0-2e56402f87f6
> Jan 18 11:12:10 localhost journal: runner INFO     162: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/d77ad2c9-ff01-4da8-b756-7eb9082f8a46/runs/d582fa19-7969-45a4-af3c-0c7e64565c73
> Jan 18 11:12:21 localhost journal: runner INFO     3833: Running
> https://129.22.159.119/pscheduler/tasks/3fe3f347-27b8-4a6f-94bd-cfdd65da4d83/runs/5c2e9bf8-6745-4e5c-b502-fbd9d6f1735d
> Jan 18 11:12:21 localhost journal: runner INFO     3833: With
> traceroute: trace --dest 129.22.152.78 --ip-version 4 --source
> 129.22.159.119 --source-node 129.22.159.119
> Jan 18 11:12:21 localhost journal: runner INFO     3833: Run succeeded.
> Jan 18 11:12:34 localhost journal: runner INFO     264: Running
> https://129.22.159.119/pscheduler/tasks/ec07f448-1407-4be4-9303-590cc635bc81/runs/85567a82-8168-49f9-8e90-fe268c25b55b
> Jan 18 11:12:34 localhost journal: runner INFO     264: With ping: rtt
> --dest www.google.com --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:12:38 localhost journal: runner INFO     264: Run succeeded.
> Jan 18 11:12:45 localhost journal: runner INFO     146: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7fb6555e-e0b4-4b13-8c5a-7eb904ce08ac/runs/bf308db0-496a-48ba-8b3a-6cdb552330f3
> Jan 18 11:12:51 localhost journal: runner INFO     3830: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7676e147-2c5c-4daa-94b4-f9e2f8fb56e1/runs/da71d5f4-2107-4562-beaf-3e8e4e90d42c
> Jan 18 11:12:55 localhost journal: runner INFO     733: Running
> https://129.22.159.119/pscheduler/tasks/9190af15-c9ce-4a16-8d9d-6c6f25a1f865/runs/2669e151-beb8-4519-888c-156adee4e447
> Jan 18 11:12:55 localhost journal: runner INFO     733: With ping: rtt
> --dest 129.22.255.254 --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:12:58 localhost journal: runner INFO     174: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/b8bdcb8d-9c8e-4ce3-ae0e-64af9c3e03b1/runs/0b9f0cf8-118e-4949-8ad1-bdf844832d43
> Jan 18 11:12:58 localhost journal: runner INFO     159: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/e9872ff1-073e-444a-a00b-8dcee282d703/runs/b2b4e7a0-fd01-4514-840d-c0b9a6661dae
> Jan 18 11:12:59 localhost journal: runner INFO     733: Run succeeded.
> Jan 18 11:12:59 localhost journal: runner INFO     3832: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/ba349cad-f10e-40b9-b8b3-f802a3fd9a60/runs/4c72e3fb-c7fb-488a-89a6-ab59761b9360
> Jan 18 11:13:10 localhost journal: runner INFO     299: Running
> https://129.22.159.119/pscheduler/tasks/401e5f50-0a0d-4b9d-a3e9-3a8a04752424/runs/eacf8c30-02eb-418f-b509-481288564292
> Jan 18 11:13:10 localhost journal: runner INFO     299: With
> traceroute: trace --dest 192.5.110.70 --ip-version 4 --source
> 129.22.159.119 --source-node 129.22.159.119
> Jan 18 11:13:10 localhost journal: runner INFO     162: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/d77ad2c9-ff01-4da8-b756-7eb9082f8a46/runs/69d86e4f-afa5-4fa2-8b8b-86b81f3800d2
> Jan 18 11:13:12 localhost journal: runner INFO     722: Running
> https://129.22.159.119/pscheduler/tasks/c1d31f6d-0148-401b-ade4-158b8bacf380/runs/0df66bb2-6572-4a3a-aed1-e7060b4c1882
> Jan 18 11:13:12 localhost journal: runner INFO     722: With ping: rtt
> --dest 129.22.159.1 --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:13:15 localhost journal: runner INFO     299: Run succeeded.
> Jan 18 11:13:16 localhost journal: runner INFO     722: Run succeeded.
> Jan 18 11:13:33 localhost journal: runner INFO     677: Running
> https://129.22.159.119/pscheduler/tasks/bdecde65-d0e7-4a4a-886d-62313179efa0/runs/e4320c38-2515-481b-9593-a270c5abd0ec
> Jan 18 11:13:33 localhost journal: runner INFO     677: With ping: rtt
> --dest 129.22.4.1 --ip-version 4 --source 129.22.159.119 --source-node
> 129.22.159.119 --ttl 255
> Jan 18 11:13:37 localhost journal: runner INFO     677: Run succeeded.
> Jan 18 11:13:46 localhost journal: runner INFO     146: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7fb6555e-e0b4-4b13-8c5a-7eb904ce08ac/runs/97b9f594-b6a7-4fc0-8731-8c54a05878e7
> Jan 18 11:13:51 localhost journal: runner INFO     3830: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7676e147-2c5c-4daa-94b4-f9e2f8fb56e1/runs/72396590-10e0-4383-b4a5-537be3df4f52
> Jan 18 11:13:58 localhost journal: runner INFO     174: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/b8bdcb8d-9c8e-4ce3-ae0e-64af9c3e03b1/runs/aab0181d-ce5f-410a-8535-f38e56a618c6
> Jan 18 11:13:58 localhost journal: runner INFO     159: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/e9872ff1-073e-444a-a00b-8dcee282d703/runs/47844ea4-6eb7-440b-847c-19f47a02f679
> Jan 18 11:13:59 localhost journal: runner INFO     3832: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/ba349cad-f10e-40b9-b8b3-f802a3fd9a60/runs/1893252e-dbf2-4b65-9f88-0fc56b9d9204
> Jan 18 11:14:09 localhost journal: runner INFO     297: Running
> https://129.22.159.119/pscheduler/tasks/74215f2e-655b-428c-a241-32a9428a7ce6/runs/fded1cd4-bdd9-43fd-8a43-4355852674c1
> Jan 18 11:14:09 localhost journal: runner INFO     297: With ping: rtt
> --dest awsportal.case.edu --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:14:10 localhost journal: runner INFO     690: Running
> https://129.22.159.119/pscheduler/tasks/23884756-054e-4470-9e2f-d0d591d585bd/runs/deebe3ec-5553-49dc-84d6-d0d22becee23
> Jan 18 11:14:10 localhost journal: runner INFO     690: With ping: rtt
> --dest 129.22.105.54 --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:14:10 localhost journal: runner INFO     162: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/d77ad2c9-ff01-4da8-b756-7eb9082f8a46/runs/3f395605-5f72-4a59-b937-690a85df5088
> Jan 18 11:14:13 localhost journal: runner INFO     297: Run succeeded.
> Jan 18 11:14:14 localhost journal: runner INFO     690: Run succeeded.
> Jan 18 11:14:46 localhost journal: runner INFO     146: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7fb6555e-e0b4-4b13-8c5a-7eb904ce08ac/runs/42395129-8f8d-4857-b4b1-eab7bc66db06
> Jan 18 11:14:51 localhost journal: runner INFO     3830: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7676e147-2c5c-4daa-94b4-f9e2f8fb56e1/runs/432658b8-24ae-4f24-a47e-aeac3815e9b8
> Jan 18 11:14:58 localhost journal: runner INFO     174: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/b8bdcb8d-9c8e-4ce3-ae0e-64af9c3e03b1/runs/5c1976e2-2c2f-4d09-98e4-04f8ca166dc6
> Jan 18 11:14:58 localhost journal: runner INFO     159: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/e9872ff1-073e-444a-a00b-8dcee282d703/runs/aa91e59f-24b5-404d-9972-e6f294f45410
> Jan 18 11:14:59 localhost journal: runner INFO     3832: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/ba349cad-f10e-40b9-b8b3-f802a3fd9a60/runs/bb8989b5-3126-4ca6-beca-6bb05996f13a
> Jan 18 11:15:10 localhost journal: runner INFO     162: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/d77ad2c9-ff01-4da8-b756-7eb9082f8a46/runs/c9cd2e6b-b2cf-48e4-8ae5-69c2c3f59b32
> Jan 18 11:15:20 localhost journal: runner INFO     301: Running
> https://129.22.159.119/pscheduler/tasks/31e0b510-78ea-49e6-96f4-652abffa7dd5/runs/09d4c0d9-694c-4444-a804-c18b85e6d25f
> Jan 18 11:15:20 localhost journal: runner INFO     301: With
> traceroute: trace --dest perfsonar.sci.cwru.edu --ip-version 4
> --source 129.22.159.119 --source-node 129.22.159.119
> Jan 18 11:15:25 localhost journal: runner INFO     301: Run succeeded.
> Jan 18 11:15:31 localhost journal: runner INFO     248: Running
> https://129.22.159.119/pscheduler/tasks/c5c06196-0704-4416-8ed3-b24135ac8489/runs/06aeceef-ca9d-40fc-aa0d-cafe2c70235c
> Jan 18 11:15:31 localhost journal: runner INFO     248: With
> traceroute: trace --dest 129.22.255.254 --ip-version 4 --source
> 129.22.159.119 --source-node 129.22.159.119
> Jan 18 11:15:31 localhost journal: runner INFO     248: Run succeeded.
> Jan 18 11:15:46 localhost journal: runner INFO     146: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7fb6555e-e0b4-4b13-8c5a-7eb904ce08ac/runs/9ead49d6-8118-4ec7-ba69-2a09bab97894
> Jan 18 11:15:51 localhost journal: runner INFO     3830: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7676e147-2c5c-4daa-94b4-f9e2f8fb56e1/runs/b3e635dd-1653-442c-9676-e4dc2261902a
> Jan 18 11:15:53 localhost journal: runner INFO     314: Running
> https://129.22.159.119/pscheduler/tasks/74215f2e-655b-428c-a241-32a9428a7ce6/runs/363775b1-e8ff-4f2b-b49c-bb4bef100b61
> Jan 18 11:15:53 localhost journal: runner INFO     314: With ping: rtt
> --dest awsportal.case.edu --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:15:54 localhost journal: runner INFO     331: Running
> https://129.22.159.119/pscheduler/tasks/a7b5bfcd-f6d2-4860-b7fd-f73703353b24/runs/2cc6c0d5-d6a0-4097-a375-f1f2628394d7
> Jan 18 11:15:54 localhost journal: runner INFO     331: With
> traceroute: trace --dest www.google.com --ip-version 4 --source
> 129.22.159.119 --source-node 129.22.159.119
> Jan 18 11:15:54 localhost journal: runner INFO     331: Run succeeded.
> Jan 18 11:15:57 localhost journal: runner INFO     314: Run succeeded.
> Jan 18 11:15:58 localhost journal: runner INFO     174: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/b8bdcb8d-9c8e-4ce3-ae0e-64af9c3e03b1/runs/0cb27eba-57a2-48f5-a4fe-55abe64253a4
> Jan 18 11:15:58 localhost journal: runner INFO     159: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/e9872ff1-073e-444a-a00b-8dcee282d703/runs/a5e847c5-629a-42da-b4ad-240b50a7cda2
> Jan 18 11:15:59 localhost journal: runner INFO     3832: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/ba349cad-f10e-40b9-b8b3-f802a3fd9a60/runs/ac4711ea-7d2c-4d97-842f-917b09834751
> Jan 18 11:16:10 localhost journal: runner INFO     162: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/d77ad2c9-ff01-4da8-b756-7eb9082f8a46/runs/b8ec93b1-2cdd-41fc-9a2a-afb09486917a
> Jan 18 11:16:23 localhost journal: runner INFO     349: Running
> https://129.22.159.119/pscheduler/tasks/9135a851-c6d8-4bd8-87cd-e2b0c33c5dc7/runs/d7a85924-73fd-433c-842b-2ba8ebe180ab
> Jan 18 11:16:23 localhost journal: runner INFO     349: With
> traceroute: trace --dest 129.22.105.54 --ip-version 4 --source
> 129.22.159.119 --source-node 129.22.159.119
> Jan 18 11:16:23 localhost journal: runner INFO     349: Run succeeded.
> Jan 18 11:16:24 localhost journal: runner INFO     319: Running
> https://129.22.159.119/pscheduler/tasks/621269ac-75dd-4120-9d41-42372141e988/runs/ccfd149a-b972-4c14-846d-5f41598e9fcc
> Jan 18 11:16:24 localhost journal: runner INFO     319: With ping: rtt
> --dest www.stanford.edu --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:16:28 localhost journal: runner INFO     319: Run succeeded.
> Jan 18 11:16:32 localhost journal: runner INFO     281: Running
> https://129.22.159.119/pscheduler/tasks/ec07f448-1407-4be4-9303-590cc635bc81/runs/13099b77-0810-474a-847e-7e30804bab34
> Jan 18 11:16:32 localhost journal: runner INFO     281: With ping: rtt
> --dest www.google.com --ip-version 4 --source 129.22.159.119
> --source-node 129.22.159.119 --ttl 255
> Jan 18 11:16:36 localhost journal: runner INFO     281: Run succeeded.
> Jan 18 11:16:46 localhost journal: runner INFO     146: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7fb6555e-e0b4-4b13-8c5a-7eb904ce08ac/runs/0a374974-c18e-4b7b-9824-012cb35dbd2d
> Jan 18 11:16:48 localhost journal: runner INFO     3834: Running
> https://129.22.159.119/pscheduler/tasks/3fe3f347-27b8-4a6f-94bd-cfdd65da4d83/runs/8e712beb-f3ae-41bc-9b8a-d190afd84186
> Jan 18 11:16:48 localhost journal: runner INFO     3834: With
> traceroute: trace --dest 129.22.152.78 --ip-version 4 --source
> 129.22.159.119 --source-node 129.22.159.119
> Jan 18 11:16:48 localhost journal: runner INFO     3834: Run succeeded.
> Jan 18 11:16:56 localhost journal: runner INFO     3830: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/7676e147-2c5c-4daa-94b4-f9e2f8fb56e1/runs/4be26abb-3b14-4e0d-bd39-9f31b73e8734
> Jan 18 11:16:58 localhost journal: runner INFO     174: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/b8bdcb8d-9c8e-4ce3-ae0e-64af9c3e03b1/runs/9c3b446a-a22a-4845-bb5e-865821665ca4
> Jan 18 11:16:58 localhost journal: runner INFO     159: Posted result
> to 
> https://129.22.159.119/pscheduler/tasks/e9872ff1-073e-444a-a00b-8dcee282d703/runs/0b1c36db-5cbd-4ae3-a11e-6804c84aac0e

> --
> To unsubscribe from this list: 
> https://lists.internet2.edu/sympa/signoff/perfsonar-user


--
Michael Johnson
GlobalNOC DevOps Engineer

Attachment: smime.p7s
Description: S/MIME cryptographic signature