perfSONAR User Q&A and Other Discussion

[Tony Plovich <>]

Hello Andy,

I've attached the JSON that's returned from http://localhost/esmond/perfsonar/archive, it appears to be the results of a throughput test, although I don't see any information on when it was conducted so I don't know if it's current.  Looking through /var/log/esmond/esmond.log I see the following:

2017-09-28 10:18:07,433 [INFO] /usr/lib/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: Connection 139641070980560 (localhost:9160) in pool 139641070979088 failed: TSocket read 0 bytes
2017-09-28 10:18:12,562 [INFO] /usr/lib/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: Connection 139641070980560 (localhost:9160) in pool 139641070979088 failed: TSocket read 0 bytes

One thing to note is that we've customized the system for our environment by disabling nscd and enabling sssd.

--
Tony Plovich ()
Business Information Systems (BIS)
Argonne National Laboratory
Phone: +1 630.252.2359

On 09/29/2017 09:16 AM, Andrew Lake wrote:

Hi,

What do you get when you go to  http://localhost/esmond/perfsonar/archive? Anything in the logs under /var/log/esmond?

Thanks,

Andy

On September 29, 2017 at 7:25:54 AM, Marian Babik () wrote:

Just to add that I have seen the same issue on several boxes already, so it must be something more generic.

Cheers,
Marian

> On Sep 28, 2017, at 6:47 PM, Tony Plovich <> wrote:
>
> Hello,
>
> I'm trying to figure out an issue between pscheduler and esmond, specifically I'm seeing the following in our /var/log/pscheduler/pscheduler.log file, and I'm not getting any graphed results for our tests:
>
> Sep 28 11:30:53 anlborder-ps journal: runner INFO 12142: With traceroute: trace --dest wash-pt1.es.net --probe-type icmp --source anlborder-ps.it.anl.gov
> Sep 28 11:30:55 anlborder-ps journal: runner INFO 12280: Running https://130.202.222.58/pscheduler/tasks/3a5c8b45-8e84-4031-ae2b-0ee7ca32342c/runs/b113c256-6a60-4a52-a939-a0af45348199
> Sep 28 11:30:55 anlborder-ps journal: runner INFO 12280: With iperf3: throughput --duration PT20S --source 130.202.222.58 --ip-version 4 --dest 164.54.59.69 --parallel 1
> Sep 28 11:30:57 anlborder-ps journal: runner INFO 12142: Run succeeded.
> Sep 28 11:31:18 anlborder-ps journal: runner INFO 12280: Run succeeded.
> Sep 28 11:31:45 anlborder-ps journal: runner WARNING 12280: Unable to update run https://164.54.59.69/pscheduler/tasks/3a5c8b45-8e84-4031-ae2b-0ee7ca32342c/runs/b113c256-6a60-4a52-a939-a0af45348199: 400 Connection error: ('Connection aborted.', BadStatusLine("''",))
> Sep 28 11:33:08 anlborder-ps journal: runner INFO 12143: Running https://anlborder-ps.it.anl.gov/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab
> Sep 28 11:33:08 anlborder-ps journal: runner INFO 12143: With traceroute: trace --dest sunn-pt1.es.net --probe-type icmp --source anlborder-ps.it.anl.gov
> Sep 28 11:33:08 anlborder-ps journal: runner INFO 12143: Run succeeded.
> Sep 28 11:33:09 anlborder-ps journal: archiver WARNING 11263: Failed to archive https://localhost/pscheduler/tasks/da3246a3-9ccb-4629-844a-c133bc884ba4/runs/ba897886-c7dd-4246-8265-8cfc42f4e1cf to esmond: 500: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11630: Failed to archive https://localhost/pscheduler/tasks/7620a622-b747-49ec-a2c6-13b28668076a/runs/92c3276e-e09d-4539-b20c-075c71e7780c to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11630: Gave up archiving https://localhost/pscheduler/tasks/7620a622-b747-49ec-a2c6-13b28668076a/runs/92c3276e-e09d-4539-b20c-075c71e7780c to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11261: Failed to archive https://localhost/pscheduler/tasks/99d558a9-717e-4b61-bb97-80a822bb5661/runs/0833d56b-1d0f-42ca-bfcd-e18f58dd2af4 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11002: Failed to archive https://localhost/pscheduler/tasks/a0da76c7-15ab-412b-bdea-e318c2657bee/runs/422618ac-b011-4c3c-852e-5454a604ed10 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 10632: Failed to archive https://localhost/pscheduler/tasks/99d558a9-717e-4b61-bb97-80a822bb5661/runs/ef05d2f3-3246-49f9-8325-0e22b92103c7 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 10818: Failed to archive https://localhost/pscheduler/tasks/a7af72dc-6d54-4b80-8c06-2c1323063f81/runs/1294280b-2403-4df2-a6ef-141700bfa299 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11633: Failed to archive https://localhost/pscheduler/tasks/70c0ae6a-ce80-4489-ab83-743580e898ee/runs/895360ac-847e-4541-aba4-fb89d5a84267 to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11633: Gave up archiving https://localhost/pscheduler/tasks/70c0ae6a-ce80-4489-ab83-743580e898ee/runs/895360ac-847e-4541-aba4-fb89d5a84267 to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11638: Failed to archive https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11632: Failed to archive https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/6395ff61-329f-4019-8c0a-c6c16faac4dd to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11632: Gave up archiving https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/6395ff61-329f-4019-8c0a-c6c16faac4dd to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 10810: Failed to archive https://localhost/pscheduler/tasks/a25e0fa2-16f7-43f8-8381-bee70ba7394c/runs/d8b498b8-b662-4b62-ba60-9460af77781e to esmond: 400: Invalid JSON returned
> Sep 28 11:34:17 anlborder-ps journal: archiver WARNING 11635: Failed to archive https://localhost/pscheduler/tasks/9c6b8a4c-8c94-41f7-a9d9-e4eeb5682ae7/runs/cb39fb6e-c616-432d-b6ae-8e5783c50b8d to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:34:17 anlborder-ps journal: archiver WARNING 11635: Gave up archiving https://localhost/pscheduler/tasks/9c6b8a4c-8c94-41f7-a9d9-e4eeb5682ae7/runs/cb39fb6e-c616-432d-b6ae-8e5783c50b8d to esmond
> Sep 28 11:34:22 anlborder-ps journal: archiver WARNING 11638: Failed to archive https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab to esmond: 400: Invalid JSON returned
>
> The JSON is in the logged URLs appears to be good and reports successful test results (hostname is anlborder-ps.it.anl.gov) so it looks like esmond is returning a bad result when pscheduler tries to store the results. I tried the following test to check on esmond, but got a 404:
>
> $ curl -v http://localhost/v1/oidset/
> * About to connect() to localhost port 80 (#0)
> * Trying ::1...
> * Connected to localhost (::1) port 80 (#0)
> > GET /v1/oidset/ HTTP/1.1
> > User-Agent: curl/7.29.0
> > Host: localhost
> > Accept: */*
> >
> < HTTP/1.1 404 Not Found
> < Date: Thu, 28 Sep 2017 16:44:20 GMT
> < Server: Apache
> < Content-Length: 208
> < Connection: close
> < Content-Type: text/html; charset=iso-8859-1
> <
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>404 Not Found</title>
> </head><body>
> <h1>Not Found</h1>
> <p>The requested URL /v1/oidset/ was not found on this server.</p>
> </body></html>
> * Closing connection 0
>
> Are there any other things I can look at in order to troubleshoot this?
>
> Thanks,
>
> --
> Tony Plovich ()
> Business Information Systems (BIS)
> Argonne National Laboratory
> Phone: +1 630.252.2359
>

Attachment: esmond_returned.json
Description: application/json