perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

Hi Szymon,

cassandra/cassandra.log and cassandra/system.log look similar to log files 
on other hosts in the mesh. There's nothing that jumps out as clearly 
indicating a problem

There are 11 hosts in the mesh. Each runs both throughput and latency tests 
on the same interface.

Kathy


On 1/17/2019 9:34 AM, Szymon Trocha wrote:
> W dniu 17.01.2019 o 15:11, Kathy Benninger pisze:
> > A perfSONAR built with NetInstall Toolkit v4.1.5-1.el7 from late December 
> > is reporting the following error on its homepage under "Test Results":
> >
> >   Error loading test listing; measurement archive unreachable:
> > http://192.231.244.54/esmond/perfsonar/archive/
> >
> > The pS is running tests and responds to test requests from other pSs in 
> > the same meshconfig file.
> >
> > httpd/error_log shows timeouts when I bring up the homepage, e.g.,:
> >
> > [Wed Jan 16 11:17:46.585696 2019] [cgi:warn] [pid 21553] [client 
> > 128.182.160.119:52235] AH01220: Timeout waiting for output from CGI script 
> > /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
> > http://192.231.244.54/toolkit/
> > [Wed Jan 16 11:17:46.585765 2019] [cgi:error] [pid 21553] [client 
> > 128.182.160.119:52235] Script timed out before returning headers: 
> > graphData.cgi, referer: http://192.231.244.54/toolkit/
> > [Wed Jan 16 11:21:48.043369 2019] [cgi:warn] [pid 19343] [client 
> > 128.182.160.119:52338] AH01220: Timeout waiting for output from CGI script 
> > /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
> > http://192.231.244.54/toolkit/
> > [Wed Jan 16 11:21:48.043438 2019] [cgi:error] [pid 19343] [client 
> > 128.182.160.119:52338] Script timed out before returning headers: 
> > graphData.cgi, referer: http://192.231.244.54/toolkit/
> > [Wed Jan 16 11:33:43.135517 2019] [cgi:warn] [pid 23720] [client 
> > 128.182.160.119:52536] AH01220: Timeout waiting for output from CGI script 
> > /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
> > http://192.231.244.54/toolkit/
> > [Wed Jan 16 11:33:43.135577 2019] [cgi:error] [pid 23720] [client 
> > 128.182.160.119:52536] Script timed out before returning headers: 
> > graphData.cgi, referer: http://192.231.244.54/toolkit/
> >
> > esmond/django.log:
> >
> > 2019-01-16 10:32:05,652 [ERROR] 
> > /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py: 
> > Internal Server Error: 
> > /esmond/perfsonar/archive/7c9ba2a49ac24a42833efaeec83e6ab7/
> > Traceback (most recent call last):
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py", 
> > line 42, in inner
> >     response = get_response(request)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 
> > 249, in _legacy_get_response
> >     response = self._get_response(request)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 
> > 187, in _get_response
> >     response = self.process_exception_by_middleware(e, request)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", line 
> > 185, in _get_response
> >     response = wrapped_callback(request, *callback_args, **callback_kwargs)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", 
> > line 58, in wrapped_view
> >     return view_func(*args, **kwargs)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", 
> > line 90, in view
> >     return self.dispatch(request, *args, **kwargs)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", 
> > line 489, in dispatch
> >     response = self.handle_exception(exc)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", 
> > line 449, in handle_exception
> >     self.raise_uncaught_exception(exc)
> >   File 
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", 
> > line 486, in dispatch
> >     response = handler(request, *args, **kwargs)
> >   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 887, in update
> >     check_connection()
> >   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 78, in 
> > check_connection
> >     db = CASSANDRA_DB(get_config(get_config_path()))
> >   File "/usr/lib/esmond/esmond/cassandra.py", line 134, in __init__
> >     "at %s - %s" % (config.cassandra_servers[0], e))
> > ConnectionException: "System Manager can't connect to Cassandra at 
> > localhost:9160 - Could not connect to localhost:9160"
> >
> > I've tried re-installing the Toolkit, but end up in the same situation. 
> > Oddly, a new install works correctly for ~5 minutes (i.e., all the tests 
> > are listed and linked on the homepage), but then it eventually starts 
> > returning the error that the archive is unreachable.
> >
> > Thoughts on how to debug or what could be wrong?
>
>
> Hi Kathy,
>
> is there anything special in /var/log/cassandra/ ?
>
> Are there many tests to load in your host?
>
> Regards,
>
> --
> Szymon Trocha
> Poznań Supercomputing & Networking Center
> General NOC phone +48 61-858-2015 | noc.pcss.pl <http://noc.pcss.pl>
> Personal desk phone +48 61-858-2022
> Wysłaliśmy do Ciebie ten e-mail w odpowiedzi na Twoje zapytanie lub w 
> związku z oferowaną usługą. Przesłanie korespondencji do Centrum Zarządzania 
> PCSS lub zgłoszenie telefoniczne jest równoznaczne z wyrażeniem zgody na 
> przetwarzanie danych osobowych przez Instytut Chemii Bioorganicznej Polskiej 
> Akademii Nauk w Poznaniu adres: ul. Z. Noskowskiego 12/14, 61-704 Poznań. 
> Szczegółowe informacje znajdują się w naszej Polityce prywatności 
> <http://noc.pcss.pl/index.html#PolitykaPrywatnosci>. | This message has been 
> sent as a part of communication with PSNC NOC or your service request sent 
> to us. For more information read our Privacy Policy 
> <http://noc.psnc.pl/index.html#PrivacyPolicy>.