perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

A perfSONAR built with NetInstall Toolkit v4.1.5-1.el7 from late December is 
reporting the following error on its homepage under "Test Results":

  Error loading test listing; measurement archive unreachable:
  http://192.231.244.54/esmond/perfsonar/archive/

The pS is running tests and responds to test requests from other pSs in the 
same meshconfig file.

httpd/error_log shows timeouts when I bring up the homepage, e.g.,:

[Wed Jan 16 11:17:46.585696 2019] [cgi:warn] [pid 21553] [client 
128.182.160.119:52235] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://192.231.244.54/toolkit/
[Wed Jan 16 11:17:46.585765 2019] [cgi:error] [pid 21553] [client 
128.182.160.119:52235] Script timed out before returning headers: 
graphData.cgi, referer: http://192.231.244.54/toolkit/
[Wed Jan 16 11:21:48.043369 2019] [cgi:warn] [pid 19343] [client 
128.182.160.119:52338] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://192.231.244.54/toolkit/
[Wed Jan 16 11:21:48.043438 2019] [cgi:error] [pid 19343] [client 
128.182.160.119:52338] Script timed out before returning headers: 
graphData.cgi, referer: http://192.231.244.54/toolkit/
[Wed Jan 16 11:33:43.135517 2019] [cgi:warn] [pid 23720] [client 
128.182.160.119:52536] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://192.231.244.54/toolkit/
[Wed Jan 16 11:33:43.135577 2019] [cgi:error] [pid 23720] [client 
128.182.160.119:52536] Script timed out before returning headers: 
graphData.cgi, referer: http://192.231.244.54/toolkit/

esmond/django.log:

2019-01-16 10:32:05,652 [ERROR] 
/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py: 
Internal Server Error: 
/esmond/perfsonar/archive/7c9ba2a49ac24a42833efaeec83e6ab7/
Traceback (most recent call last):
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py", 
line 42, in inner
    response = get_response(request)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 249, in _legacy_get_response
    response = self._get_response(request)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", 
line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", 
line 90, in view
    return self.dispatch(request, *args, **kwargs)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 
489, in dispatch
    response = self.handle_exception(exc)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 
449, in handle_exception
    self.raise_uncaught_exception(exc)
  File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", line 
486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 887, in update
    check_connection()
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 78, in 
check_connection
    db = CASSANDRA_DB(get_config(get_config_path()))
  File "/usr/lib/esmond/esmond/cassandra.py", line 134, in __init__
    "at %s - %s" % (config.cassandra_servers[0], e))
ConnectionException: "System Manager can't connect to Cassandra at 
localhost:9160 - Could not connect to localhost:9160"

I've tried re-installing the Toolkit, but end up in the same situation. 
Oddly, a new install works correctly for ~5 minutes (i.e., all the tests are 
listed and linked on the homepage), but then it eventually starts returning 
the error that the archive is unreachable.

Thoughts on how to debug or what could be wrong?

Thanks,
Kathy