perfSONAR User Q&A and Other Discussion

[Marian Babik <>]

Just to add that I have seen the same issue on several boxes already, so it 
must be something more generic.

Cheers,
Marian

> On Sep 28, 2017, at 6:47 PM, Tony Plovich 
> <>
>  wrote:
> 
> Hello,
> 
> I'm trying to figure out an issue between pscheduler and esmond, 
> specifically I'm seeing the following in our 
> /var/log/pscheduler/pscheduler.log file, and I'm not getting any graphed 
> results for our tests:
> 
> Sep 28 11:30:53 anlborder-ps journal: runner INFO     12142: With 
> traceroute: trace --dest wash-pt1.es.net --probe-type icmp --source 
> anlborder-ps.it.anl.gov
> Sep 28 11:30:55 anlborder-ps journal: runner INFO     12280: Running 
> https://130.202.222.58/pscheduler/tasks/3a5c8b45-8e84-4031-ae2b-0ee7ca32342c/runs/b113c256-6a60-4a52-a939-a0af45348199
> Sep 28 11:30:55 anlborder-ps journal: runner INFO     12280: With iperf3: 
> throughput --duration PT20S --source 130.202.222.58 --ip-version 4 --dest 
> 164.54.59.69 --parallel 1
> Sep 28 11:30:57 anlborder-ps journal: runner INFO     12142: Run succeeded.
> Sep 28 11:31:18 anlborder-ps journal: runner INFO     12280: Run succeeded.
> Sep 28 11:31:45 anlborder-ps journal: runner WARNING  12280: Unable to 
> update run 
> https://164.54.59.69/pscheduler/tasks/3a5c8b45-8e84-4031-ae2b-0ee7ca32342c/runs/b113c256-6a60-4a52-a939-a0af45348199:
>  400 Connection error: ('Connection aborted.', BadStatusLine("''",))
> Sep 28 11:33:08 anlborder-ps journal: runner INFO     12143: Running 
> https://anlborder-ps.it.anl.gov/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab
> Sep 28 11:33:08 anlborder-ps journal: runner INFO     12143: With 
> traceroute: trace --dest sunn-pt1.es.net --probe-type icmp --source 
> anlborder-ps.it.anl.gov
> Sep 28 11:33:08 anlborder-ps journal: runner INFO     12143: Run succeeded.
> Sep 28 11:33:09 anlborder-ps journal: archiver WARNING  11263: Failed to 
> archive 
> https://localhost/pscheduler/tasks/da3246a3-9ccb-4629-844a-c133bc884ba4/runs/ba897886-c7dd-4246-8265-8cfc42f4e1cf
>  to esmond: 500: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11630: Failed to 
> archive 
> https://localhost/pscheduler/tasks/7620a622-b747-49ec-a2c6-13b28668076a/runs/92c3276e-e09d-4539-b20c-075c71e7780c
>  to esmond: Archiver permanently abandoned registering test after 3 
> attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11630: Gave up 
> archiving 
> https://localhost/pscheduler/tasks/7620a622-b747-49ec-a2c6-13b28668076a/runs/92c3276e-e09d-4539-b20c-075c71e7780c
>  to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11261: Failed to 
> archive 
> https://localhost/pscheduler/tasks/99d558a9-717e-4b61-bb97-80a822bb5661/runs/0833d56b-1d0f-42ca-bfcd-e18f58dd2af4
>  to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11002: Failed to 
> archive 
> https://localhost/pscheduler/tasks/a0da76c7-15ab-412b-bdea-e318c2657bee/runs/422618ac-b011-4c3c-852e-5454a604ed10
>  to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  10632: Failed to 
> archive 
> https://localhost/pscheduler/tasks/99d558a9-717e-4b61-bb97-80a822bb5661/runs/ef05d2f3-3246-49f9-8325-0e22b92103c7
>  to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  10818: Failed to 
> archive 
> https://localhost/pscheduler/tasks/a7af72dc-6d54-4b80-8c06-2c1323063f81/runs/1294280b-2403-4df2-a6ef-141700bfa299
>  to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11633: Failed to 
> archive 
> https://localhost/pscheduler/tasks/70c0ae6a-ce80-4489-ab83-743580e898ee/runs/895360ac-847e-4541-aba4-fb89d5a84267
>  to esmond: Archiver permanently abandoned registering test after 3 
> attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11633: Gave up 
> archiving 
> https://localhost/pscheduler/tasks/70c0ae6a-ce80-4489-ab83-743580e898ee/runs/895360ac-847e-4541-aba4-fb89d5a84267
>  to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11638: Failed to 
> archive 
> https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab
>  to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11632: Failed to 
> archive 
> https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/6395ff61-329f-4019-8c0a-c6c16faac4dd
>  to esmond: Archiver permanently abandoned registering test after 3 
> attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  11632: Gave up 
> archiving 
> https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/6395ff61-329f-4019-8c0a-c6c16faac4dd
>  to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING  10810: Failed to 
> archive 
> https://localhost/pscheduler/tasks/a25e0fa2-16f7-43f8-8381-bee70ba7394c/runs/d8b498b8-b662-4b62-ba60-9460af77781e
>  to esmond: 400: Invalid JSON returned
> Sep 28 11:34:17 anlborder-ps journal: archiver WARNING  11635: Failed to 
> archive 
> https://localhost/pscheduler/tasks/9c6b8a4c-8c94-41f7-a9d9-e4eeb5682ae7/runs/cb39fb6e-c616-432d-b6ae-8e5783c50b8d
>  to esmond: Archiver permanently abandoned registering test after 3 
> attempt(s): 400: Invalid JSON returned
> Sep 28 11:34:17 anlborder-ps journal: archiver WARNING  11635: Gave up 
> archiving 
> https://localhost/pscheduler/tasks/9c6b8a4c-8c94-41f7-a9d9-e4eeb5682ae7/runs/cb39fb6e-c616-432d-b6ae-8e5783c50b8d
>  to esmond
> Sep 28 11:34:22 anlborder-ps journal: archiver WARNING  11638: Failed to 
> archive 
> https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab
>  to esmond: 400: Invalid JSON returned
> 
> The JSON is in the logged URLs appears to be good and reports successful 
> test results (hostname is anlborder-ps.it.anl.gov) so it looks like esmond 
> is returning a bad result when pscheduler tries to store the results.  I 
> tried the following test to check on esmond, but got a 404:
> 
> $ curl  -v http://localhost/v1/oidset/
> * About to connect() to localhost port 80 (#0)
> *   Trying ::1...
> * Connected to localhost (::1) port 80 (#0)
> > GET /v1/oidset/ HTTP/1.1
> > User-Agent: curl/7.29.0
> > Host: localhost
> > Accept: */*
> >
> < HTTP/1.1 404 Not Found
> < Date: Thu, 28 Sep 2017 16:44:20 GMT
> < Server: Apache
> < Content-Length: 208
> < Connection: close
> < Content-Type: text/html; charset=iso-8859-1
> <
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>404 Not Found</title>
> </head><body>
> <h1>Not Found</h1>
> <p>The requested URL /v1/oidset/ was not found on this server.</p>
> </body></html>
> * Closing connection 0
> 
> Are there any other things I can look at in order to troubleshoot this?
> 
> Thanks,
> 
> --
> Tony Plovich 
> ()
> Business Information Systems (BIS)
> Argonne National Laboratory
> Phone: +1 630.252.2359
>

Attachment: smime.p7s
Description: S/MIME cryptographic signature