perfSONAR User Q&A and Other Discussion

[Trond Endrestøl <>]

On Wed, 4 Oct 2017 15:55-0000, Mark Feit wrote:

> Trond Endrestøl writes:
> 
>     A freshly installed "perfSONAR 4.0.1, CentOS 7, FullInstall", displays 
>     no test results in either dashboards.
>     
>     All it says is: Error loading test listing: Internal Server Error
> 
> Can you have a look in the logs at /var/log/httpd/ssl_error_log and 
> /var/log/pscheduler/pscheduler.log and let me know what you see?  
> (Or if you’d like to just send them to me off-list, that would be 
> fine, too.  I’ll follow up on-list.)

I obfuscated the IPv6 address, and I don't mind sending the logs 
on-list in that state.

Here are the last few lines of /var/log/httpd/ssl_error_log:

[Wed Oct 04 17:47:55.309973 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180] mod_wsgi (pid=1416): Exception occurred processing WSGI 
script '/usr/lib/esmond/esmond/wsgi.py'.
[Wed Oct 04 17:47:55.310018 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180] Traceback (most recent call last):
[Wed Oct 04 17:47:55.310028 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File "/usr/lib/esmond/esmond/wsgi.py", line 28, in 
application
[Wed Oct 04 17:47:55.310071 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     return get_wsgi_application()(environ, start_response)
[Wed Oct 04 17:47:55.310078 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/wsgi.py", 
line 189, in __call__
[Wed Oct 04 17:47:55.310088 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     response = self.get_response(request)
[Wed Oct 04 17:47:55.310092 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 218, in get_response
[Wed Oct 04 17:47:55.310100 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     response = self.handle_uncaught_exception(request, 
resolver, sys.exc_info())
[Wed Oct 04 17:47:55.310104 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 264, in handle_uncaught_exception
[Wed Oct 04 17:47:55.310112 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     if resolver.urlconf_module is None:
[Wed Oct 04 17:47:55.310116 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/urlresolvers.py", 
line 395, in urlconf_module
[Wed Oct 04 17:47:55.310123 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     self._urlconf_module = import_module(self.urlconf_name)
[Wed Oct 04 17:47:55.310127 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File "/usr/lib64/python2.7/importlib/__init__.py", line 
37, in import_module
[Wed Oct 04 17:47:55.310134 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     __import__(name)
[Wed Oct 04 17:47:55.310138 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File "/usr/lib/esmond/esmond/urls.py", line 27, in 
<module>
[Wed Oct 04 17:47:55.310145 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     from esmond.api.perfsonar.api_v2 import (
[Wed Oct 04 17:47:55.310149 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", 
line 65, in <module>
[Wed Oct 04 17:47:55.310156 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180]     raise ConnectionException(error_msg)
[Wed Oct 04 17:47:55.310167 2017] [:error] [pid 1416] [remote 
2001:x:y:z::2:180] ConnectionException: 'Unable to connect to cassandra. 
Please verify cassandra is running.'

Here are the last few lines of /var/log/pscheduler.log

Oct  4 17:45:04 perfsonar-cma journal: ticker WARNING  Queue maintainer got 
exception server closed the connection unexpectedly#012#011This probably 
means the server terminated abnormally#012#011before or while processing the 
request.
Oct  4 17:45:04 perfsonar-cma journal: safe_run/archiver ERROR    Program 
threw an exception after 2:00:12.199236
Oct  4 17:45:04 perfsonar-cma journal: safe_run/runner ERROR    Program threw 
an exception after 2:00:12.198754
Oct  4 17:45:04 perfsonar-cma journal: safe_run/scheduler ERROR    Program 
threw an exception after 2:00:12.198775
Oct  4 17:45:04 perfsonar-cma journal: safe_run/runner ERROR    Exception: 
DatabaseError: server closed the connection unexpectedly#012#011This probably 
means the server terminated abnormally#012#011before or while processing the 
request.#012#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 41, in 
safe_run#012    function()#012  File 
"/usr/libexec/pscheduler/daemons/runner", line 907, in <lambda>#012    
pscheduler.safe_run(lambda: main_program())#012  File 
"/usr/libexec/pscheduler/daemons/runner", line 817, in main_program#012    
""", [refresh]);#012DatabaseError: server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Oct  4 17:45:04 perfsonar-cma journal: safe_run/runner ERROR    Waiting 0.25 
seconds before restarting
Oct  4 17:45:04 perfsonar-cma journal: safe_run/scheduler ERROR    Exception: 
DatabaseError: server closed the connection unexpectedly#012#011This probably 
means the server terminated abnormally#012#011before or while processing the 
request.#012#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 41, in 
safe_run#012    function()#012  File 
"/usr/libexec/pscheduler/daemons/scheduler", line 793, in <lambda>#012    
pscheduler.safe_run(lambda: main_program())#012  File 
"/usr/libexec/pscheduler/daemons/scheduler", line 749, in main_program#012    
cursor.execute(query, args)#012DatabaseError: server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Oct  4 17:45:04 perfsonar-cma journal: safe_run/scheduler ERROR    Waiting 
0.25 seconds before restarting
Oct  4 17:45:04 perfsonar-cma journal: safe_run/archiver ERROR    Exception: 
DatabaseError: server closed the connection unexpectedly#012#011This probably 
means the server terminated abnormally#012#011before or while processing the 
request.#012#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 41, in 
safe_run#012    function()#012  File 
"/usr/libexec/pscheduler/daemons/archiver", line 522, in <lambda>#012    
pscheduler.safe_run(lambda: main_program())#012  File 
"/usr/libexec/pscheduler/daemons/archiver", line 478, in main_program#012    
[options.max_parallel])#012  File 
"/usr/lib/python2.7/site-packages/pscheduler/db.py", line 187, in query#012   
 cursor.execute(query, args)#012DatabaseError: server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Oct  4 17:45:04 perfsonar-cma journal: safe_run/archiver ERROR    Waiting 
0.25 seconds before restarting
Oct  4 17:45:04 perfsonar-cma journal: safe_run/scheduler ERROR   Restarting
Oct  4 17:45:04 perfsonar-cma journal: safe_run/runner ERROR    Restarting
Oct  4 17:45:04 perfsonar-cma journal: safe_run/archiver ERROR    Restarting
Oct  4 17:45:16 perfsonar-cma journal: safe_run/ticker ERROR    Program threw 
an exception after 2:00:24.042318
Oct  4 17:45:16 perfsonar-cma journal: safe_run/ticker ERROR    Exception: 
OperationalError: terminating connection due to administrator 
command#012server closed the connection unexpectedly#012#011This probably 
means the server terminated abnormally#012#011before or while processing the 
request.#012#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/site-packages/pscheduler/saferun.py", line 41, in 
safe_run#012    function()#012  File 
"/usr/libexec/pscheduler/daemons/ticker", line 153, in <lambda>#012    
pscheduler.safe_run(lambda: main_program())#012  File 
"/usr/libexec/pscheduler/daemons/ticker", line 133, in main_program#012    
cursor.execute("SELECT ticker()")#012OperationalError: terminating connection 
due to administrator command#012server closed the connection 
unexpectedly#012#011This probably means the server terminated 
abnormally#012#011before or while processing the request.
Oct  4 17:45:16 perfsonar-cma journal: safe_run/ticker ERROR    Waiting 0.25 
seconds before restarting
Oct  4 17:45:16 perfsonar-cma journal: safe_run/ticker ERROR    Restarting

-- 
Trond.