perfSONAR User Q&A and Other Discussion

[Szymon Trocha <>]

W dniu 17.01.2019 o 15:11, Kathy Benninger pisze:

A perfSONAR built with NetInstall Toolkit v4.1.5-1.el7 from late December is reporting the following error on its homepage under "Test Results":

  Error loading test listing; measurement archive unreachable:
  http://192.231.244.54/esmond/perfsonar/archive/

The pS is running tests and responds to test requests from other pSs in the same meshconfig file.

httpd/error_log shows timeouts when I bring up the homepage, e.g.,:

[Wed Jan 16 11:17:46.585696 2019] [cgi:warn] [pid 21553] [client 128.182.160.119:52235] AH01220: Timeout waiting for output from CGI script /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: http://192.231.244.54/toolkit/
[Wed Jan 16 11:17:46.585765 2019] [cgi:error] [pid 21553] [client 128.182.160.119:52235] Script timed out before returning headers: graphData.cgi, referer: http://192.231.244.54/toolkit/
[Wed Jan 16 11:21:48.043369 2019] [cgi:warn] [pid 19343] [client 128.182.160.119:52338] AH01220: Timeout waiting for output from CGI script /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: http://192.231.244.54/toolkit/
[Wed Jan 16 11:21:48.043438 2019] [cgi:error] [pid 19343] [client 128.182.160.119:52338] Script timed out before returning headers: graphData.cgi, referer: http://192.231.244.54/toolkit/
[Wed Jan 16 11:33:43.135517 2019] [cgi:warn] [pid 23720] [client 128.182.160.119:52536] AH01220: Timeout waiting for output from CGI script /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: http://192.231.244.54/toolkit/
[Wed Jan 16 11:33:43.135577 2019] [cgi:error] [pid 23720] [client 128.182.160.119:52536] Script timed out before returning headers: graphData.cgi, referer: http://192.231.244.54/toolkit/

esmond/django.log:

2019-01-16 10:32:05,652 [ERROR] /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py: Internal Server Error: /esmond/perfsonar/archive/7c9ba2a49ac24a42833efaeec83e6ab7/
Traceback (most recent call last):
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = get_response(request)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", line 90, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 887, in update
    check_connection()
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 78, in check_connection
    db = CASSANDRA_DB(get_config(get_config_path()))
  File "/usr/lib/esmond/esmond/cassandra.py", line 134, in __init__
    "at %s - %s" % (config.cassandra_servers[0], e))
ConnectionException: "System Manager can't connect to Cassandra at localhost:9160 - Could not connect to localhost:9160"

I've tried re-installing the Toolkit, but end up in the same situation. Oddly, a new install works correctly for ~5 minutes (i.e., all the tests are listed and linked on the homepage), but then it eventually starts returning the error that the archive is unreachable.

Thoughts on how to debug or what could be wrong?

Hi Kathy,

is there anything special in /var/log/cassandra/ ?

Are there many tests to load in your host?

Regards,

--

Szymon Trocha
Poznań Supercomputing & Networking Center

General NOC phone +48 61-858-2015 | noc.pcss.pl
Personal desk phone +48 61-858-2022

Wysłaliśmy do Ciebie ten e-mail w odpowiedzi na Twoje zapytanie lub w związku z oferowaną usługą. Przesłanie korespondencji do Centrum Zarządzania PCSS lub zgłoszenie telefoniczne jest równoznaczne z wyrażeniem zgody na przetwarzanie danych osobowych przez Instytut Chemii Bioorganicznej Polskiej Akademii Nauk w Poznaniu adres: ul. Z. Noskowskiego 12/14, 61-704 Poznań. Szczegółowe informacje znajdują się w naszej Polityce prywatności. | This message has been sent as a part of communication with PSNC NOC or your service request sent to us. For more information read our Privacy Policy.