perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi Tony,

Actually your esmond instance and tests created by your host is fine as far as I can tell. It looks like the errors are actually from tests created by other hosts testing to your box. The way pscheduler works is that the source of the test is the one responsible for archiving the results of throughput tests. For example, you have a test created by a SLAC host. It looks like psnr-bw01.slac.stanford.edu created a test where your host is the source, psnr-bw01.slac.stanford.edu is the dest and it wants the result registered to http://psnr-bw01.slac.stanford.edu/esmond/perfsonar/archive/. My guess is either or both of you have a firewall or something blocking that from happening hence the error message. You can see this in the archivings section of https://anlborder-ps.it.anl.gov/pscheduler/tasks/5158d89c-3f67-440c-b682-72b8c9bb9be7/runs/0f3961ea-ad8f-4dab-ba94-53f71b915e1b?pretty if you like to dig through JSON.

It looks like you have tests created by others in the same boat. You can click through the links in the log error messages and look at the “measurement-agent” field to see who created the test and the archiver “url” field to see where they want the result sent. 

Hope that helps,

Andy

On September 29, 2017 at 11:07:44 AM, Tony Plovich () wrote:

Hello Andy,

I've attached the JSON that's returned from http://localhost/esmond/perfsonar/archive, it appears to be the results of a throughput test, although I don't see any information on when it was conducted so I don't know if it's current.  Looking through /var/log/esmond/esmond.log I see the following:

2017-09-28 10:18:07,433 [INFO] /usr/lib/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: Connection 139641070980560 (localhost:9160) in pool 139641070979088 failed: TSocket read 0 bytes
2017-09-28 10:18:12,562 [INFO] /usr/lib/esmond/lib/python2.7/site-packages/pycassa/logging/pool_logger.py: Connection 139641070980560 (localhost:9160) in pool 139641070979088 failed: TSocket read 0 bytes

One thing to note is that we've customized the system for our environment by disabling nscd and enabling sssd.

--
Tony Plovich ()
Business Information Systems (BIS)
Argonne National Laboratory
Phone: +1 630.252.2359

On 09/29/2017 09:16 AM, Andrew Lake wrote:

Hi,

What do you get when you go to  http://localhost/esmond/perfsonar/archive? Anything in the logs under /var/log/esmond?

Thanks,

Andy

On September 29, 2017 at 7:25:54 AM, Marian Babik () wrote:

Just to add that I have seen the same issue on several boxes already, so it must be something more generic.

Cheers,
Marian

> On Sep 28, 2017, at 6:47 PM, Tony Plovich <> wrote:
>
> Hello,
>
> I'm trying to figure out an issue between pscheduler and esmond, specifically I'm seeing the following in our /var/log/pscheduler/pscheduler.log file, and I'm not getting any graphed results for our tests:
>
> Sep 28 11:30:53 anlborder-ps journal: runner INFO 12142: With traceroute: trace --dest wash-pt1.es.net --probe-type icmp --source anlborder-ps.it.anl.gov
> Sep 28 11:30:55 anlborder-ps journal: runner INFO 12280: Running https://130.202.222.58/pscheduler/tasks/3a5c8b45-8e84-4031-ae2b-0ee7ca32342c/runs/b113c256-6a60-4a52-a939-a0af45348199
> Sep 28 11:30:55 anlborder-ps journal: runner INFO 12280: With iperf3: throughput --duration PT20S --source 130.202.222.58 --ip-version 4 --dest 164.54.59.69 --parallel 1
> Sep 28 11:30:57 anlborder-ps journal: runner INFO 12142: Run succeeded.
> Sep 28 11:31:18 anlborder-ps journal: runner INFO 12280: Run succeeded.
> Sep 28 11:31:45 anlborder-ps journal: runner WARNING 12280: Unable to update run https://164.54.59.69/pscheduler/tasks/3a5c8b45-8e84-4031-ae2b-0ee7ca32342c/runs/b113c256-6a60-4a52-a939-a0af45348199: 400 Connection error: ('Connection aborted.', BadStatusLine("''",))
> Sep 28 11:33:08 anlborder-ps journal: runner INFO 12143: Running https://anlborder-ps.it.anl.gov/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab
> Sep 28 11:33:08 anlborder-ps journal: runner INFO 12143: With traceroute: trace --dest sunn-pt1.es.net --probe-type icmp --source anlborder-ps.it.anl.gov
> Sep 28 11:33:08 anlborder-ps journal: runner INFO 12143: Run succeeded.
> Sep 28 11:33:09 anlborder-ps journal: archiver WARNING 11263: Failed to archive https://localhost/pscheduler/tasks/da3246a3-9ccb-4629-844a-c133bc884ba4/runs/ba897886-c7dd-4246-8265-8cfc42f4e1cf to esmond: 500: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11630: Failed to archive https://localhost/pscheduler/tasks/7620a622-b747-49ec-a2c6-13b28668076a/runs/92c3276e-e09d-4539-b20c-075c71e7780c to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11630: Gave up archiving https://localhost/pscheduler/tasks/7620a622-b747-49ec-a2c6-13b28668076a/runs/92c3276e-e09d-4539-b20c-075c71e7780c to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11261: Failed to archive https://localhost/pscheduler/tasks/99d558a9-717e-4b61-bb97-80a822bb5661/runs/0833d56b-1d0f-42ca-bfcd-e18f58dd2af4 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11002: Failed to archive https://localhost/pscheduler/tasks/a0da76c7-15ab-412b-bdea-e318c2657bee/runs/422618ac-b011-4c3c-852e-5454a604ed10 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 10632: Failed to archive https://localhost/pscheduler/tasks/99d558a9-717e-4b61-bb97-80a822bb5661/runs/ef05d2f3-3246-49f9-8325-0e22b92103c7 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 10818: Failed to archive https://localhost/pscheduler/tasks/a7af72dc-6d54-4b80-8c06-2c1323063f81/runs/1294280b-2403-4df2-a6ef-141700bfa299 to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11633: Failed to archive https://localhost/pscheduler/tasks/70c0ae6a-ce80-4489-ab83-743580e898ee/runs/895360ac-847e-4541-aba4-fb89d5a84267 to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11633: Gave up archiving https://localhost/pscheduler/tasks/70c0ae6a-ce80-4489-ab83-743580e898ee/runs/895360ac-847e-4541-aba4-fb89d5a84267 to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11638: Failed to archive https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab to esmond: 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11632: Failed to archive https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/6395ff61-329f-4019-8c0a-c6c16faac4dd to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 11632: Gave up archiving https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/6395ff61-329f-4019-8c0a-c6c16faac4dd to esmond
> Sep 28 11:33:14 anlborder-ps journal: archiver WARNING 10810: Failed to archive https://localhost/pscheduler/tasks/a25e0fa2-16f7-43f8-8381-bee70ba7394c/runs/d8b498b8-b662-4b62-ba60-9460af77781e to esmond: 400: Invalid JSON returned
> Sep 28 11:34:17 anlborder-ps journal: archiver WARNING 11635: Failed to archive https://localhost/pscheduler/tasks/9c6b8a4c-8c94-41f7-a9d9-e4eeb5682ae7/runs/cb39fb6e-c616-432d-b6ae-8e5783c50b8d to esmond: Archiver permanently abandoned registering test after 3 attempt(s): 400: Invalid JSON returned
> Sep 28 11:34:17 anlborder-ps journal: archiver WARNING 11635: Gave up archiving https://localhost/pscheduler/tasks/9c6b8a4c-8c94-41f7-a9d9-e4eeb5682ae7/runs/cb39fb6e-c616-432d-b6ae-8e5783c50b8d to esmond
> Sep 28 11:34:22 anlborder-ps journal: archiver WARNING 11638: Failed to archive https://localhost/pscheduler/tasks/17ebe5dd-9a6b-4ea0-adf2-2e9bcc4ac825/runs/e37c441c-07be-444e-ac94-f948c0243fab to esmond: 400: Invalid JSON returned
>
> The JSON is in the logged URLs appears to be good and reports successful test results (hostname is anlborder-ps.it.anl.gov) so it looks like esmond is returning a bad result when pscheduler tries to store the results. I tried the following test to check on esmond, but got a 404:
>
> $ curl -v http://localhost/v1/oidset/
> * About to connect() to localhost port 80 (#0)
> * Trying ::1...
> * Connected to localhost (::1) port 80 (#0)
> > GET /v1/oidset/ HTTP/1.1
> > User-Agent: curl/7.29.0
> > Host: localhost
> > Accept: */*
> >
> < HTTP/1.1 404 Not Found
> < Date: Thu, 28 Sep 2017 16:44:20 GMT
> < Server: Apache
> < Content-Length: 208
> < Connection: close
> < Content-Type: text/html; charset=iso-8859-1
> <
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>404 Not Found</title>
> </head><body>
> <h1>Not Found</h1>
> <p>The requested URL /v1/oidset/ was not found on this server.</p>
> </body></html>
> * Closing connection 0
>
> Are there any other things I can look at in order to troubleshoot this?
>
> Thanks,
>
> --
> Tony Plovich ()
> Business Information Systems (BIS)
> Argonne National Laboratory
> Phone: +1 630.252.2359
>