perfSONAR User Q&A and Other Discussion

[Kathy Benninger <>]

Hi Michael,

Yes, I do see messages in both /var/log/httpd/ssl_error_log and 
/var/log/httpd/error_log that look relevant.

The messages mostly show up in /var/log/httpd/error_log:

[Thu Jan 24 09:11:17.119164 2019] [cgi:warn] [pid 19610] [client 
128.182.160.119:58686] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://ps-test.psc.edu/toolkit/
[Thu Jan 24 09:11:17.119232 2019] [cgi:error] [pid 19610] [client 
128.182.160.119:58686] Script timed out before returning headers: 
graphData.cgi, referer: http://ps-test.psc.edu/toolkit/
[Thu Jan 24 12:16:38.822578 2019] [:error] [pid 23576] [remote 
193.108.212.36:792] mod_wsgi (pid=23576): Exception occurred processing WSGI 
script '/usr/lib/esmond/esmond/wsgi.py'.
[Thu Jan 24 12:16:38.822805 2019] [:error] [pid 23576] [remote 
193.108.212.36:792] IOError: failed to write data
[Thu Jan 24 12:19:38.480746 2019] [:error] [pid 23578] [remote 
74.217.90.250:53280] mod_wsgi (pid=23578): Exception occurred processing 
WSGI script '/usr/lib/esmond/esmond/wsgi.py'.
[Thu Jan 24 12:19:38.480899 2019] [:error] [pid 23578] [remote 
74.217.90.250:53280] IOError: failed to write data
[Thu Jan 24 12:59:13.539014 2019] [cgi:warn] [pid 20823] [client 
129.79.9.117:50490] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://192.231.244.54/toolkit/
[Thu Jan 24 12:59:13.539099 2019] [cgi:error] [pid 20823] [client 
129.79.9.117:50490] Script timed out before returning headers: 
graphData.cgi, referer: http://192.231.244.54/toolkit/
[Thu Jan 24 13:00:13.599108 2019] [cgi:warn] [pid 20823] [client 
129.79.9.117:50490] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://192.231.244.54/toolkit/
[Thu Jan 24 13:02:20.078144 2019] [cgi:warn] [pid 20823] [client 
129.79.9.117:51462] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://192.231.244.54/toolkit/
[Thu Jan 24 13:02:20.078195 2019] [cgi:error] [pid 20823] [client 
129.79.9.117:51462] Script timed out before returning headers: 
graphData.cgi, referer: http://192.231.244.54/toolkit/
[Thu Jan 24 13:03:20.138247 2019] [cgi:warn] [pid 20823] [client 
129.79.9.117:51462] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://192.231.244.54/toolkit/
[Thu Jan 24 13:03:42.176895 2019] [cgi:warn] [pid 25084] [client 
129.79.9.117:51814] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:03:42.176994 2019] [cgi:error] [pid 25084] [client 
129.79.9.117:51814] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:04:01.185459 2019] [cgi:warn] [pid 17193] [client 
129.79.9.117:51905] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:04:01.185504 2019] [cgi:error] [pid 17193] [client 
129.79.9.117:51905] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:04:42.237067 2019] [cgi:warn] [pid 25084] [client 
129.79.9.117:51814] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:09:10.445599 2019] [cgi:warn] [pid 24566] [client 
129.79.9.117:53496] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:09:10.445674 2019] [cgi:error] [pid 24566] [client 
129.79.9.117:53496] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:20:57.460161 2019] [cgi:warn] [pid 549] [client 
193.108.212.36:45161] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:20:57.460206 2019] [cgi:error] [pid 549] [client 
193.108.212.36:45161] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:20:58.654694 2019] [cgi:warn] [pid 25083] [client 
193.108.212.36:36169] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:20:58.654748 2019] [cgi:error] [pid 25083] [client 
193.108.212.36:36169] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:21:53.389570 2019] [cgi:warn] [pid 32473] [client 
74.217.90.250:36961] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:21:53.389646 2019] [cgi:error] [pid 32473] [client 
74.217.90.250:36961] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:21:55.541377 2019] [cgi:warn] [pid 4170] [client 
74.217.90.250:48114] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:21:55.541454 2019] [cgi:error] [pid 4170] [client 
74.217.90.250:48114] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:21:58.714826 2019] [cgi:warn] [pid 25083] [client 
193.108.212.36:36169] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:21:58.881260 2019] [cgi:warn] [pid 547] [client 
74.217.90.250:54136] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:21:58.881321 2019] [cgi:error] [pid 547] [client 
74.217.90.250:54136] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:22:53.449713 2019] [cgi:warn] [pid 32473] [client 
74.217.90.250:36961] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:22:55.601555 2019] [cgi:warn] [pid 4170] [client 
74.217.90.250:48114] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:22:58.934192 2019] [cgi:warn] [pid 547] [client 
74.217.90.250:54136] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:30:25.036837 2019] [cgi:warn] [pid 8189] [client 
154.59.126.106:16889] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:30:25.036900 2019] [cgi:error] [pid 8189] [client 
154.59.126.106:16889] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:30:38.103161 2019] [cgi:warn] [pid 8190] [client 
154.59.126.106:19962] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:30:38.103214 2019] [cgi:error] [pid 8190] [client 
154.59.126.106:19962] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:30:49.040246 2019] [cgi:warn] [pid 4123] [client 
154.59.126.106:32057] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:30:49.040311 2019] [cgi:error] [pid 4123] [client 
154.59.126.106:32057] Script timed out before returning headers: graphData.cgi
[Thu Jan 24 13:31:38.163269 2019] [cgi:warn] [pid 8190] [client 
154.59.126.106:19962] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:31:49.100427 2019] [cgi:warn] [pid 4123] [client 
154.59.126.106:32057] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi
[Thu Jan 24 13:41:55.660149 2019] [:error] [pid 23577] [remote 
204.101.161.159:55328] mod_wsgi (pid=23577): Exception occurred processing 
WSGI script '/usr/lib/esmond/esmond/wsgi.py'.
[Thu Jan 24 13:41:55.660362 2019] [:error] [pid 23577] [remote 
204.101.161.159:55328] IOError: failed to write data
[Thu Jan 24 13:49:30.604437 2019] [cgi:warn] [pid 23560] [client 
128.182.160.119:59976] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
http://ps-test.psc.edu/toolkit/
[Thu Jan 24 13:49:30.604503 2019] [cgi:error] [pid 23560] [client 
128.182.160.119:59976] Script timed out before returning headers: 
graphData.cgi, referer: http://ps-test.psc.edu/toolkit/

New timeout messages are logged in error_log whenever I try to bring up the 
homepage.

Looking at /var/log/httpd/ssl_error_log, the current 
/var/log/httpd/ssl_error_log is empty, however, the previous 
/var/log/httpd/ssl_error_log-20190120 includes a few timeouts at the 
beginning and end of the file:

[Wed Jan 16 17:04:08.790427 2019] [cgi:warn] [pid 8519] [client 
128.182.160.119:62578] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
https://ps-test.psc.edu/toolkit/
[Wed Jan 16 17:04:08.790504 2019] [cgi:error] [pid 8519] [client 
128.182.160.119:62578] Script timed out before returning headers: 
graphData.cgi, referer: https://ps-test.psc.edu/toolkit/
[Wed Jan 16 17:04:59.951221 2019] [cgi:warn] [pid 7967] [client 
128.182.160.119:62592] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/web-ng/root/services/host.cgi, referer: 
https://ps-test.psc.edu/toolkit/
[Wed Jan 16 17:04:59.951289 2019] [cgi:error] [pid 7967] [client 
128.182.160.119:62592] Script timed out before returning headers: host.cgi, 
referer: https://ps-test.psc.edu/toolkit/
[Wed Jan 16 17:05:44.706199 2019] [cgi:warn] [pid 9366] [client 
128.182.160.119:62642] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
https://ps-test.psc.edu/toolkit/
[Wed Jan 16 17:05:44.706266 2019] [cgi:error] [pid 9366] [client 
128.182.160.119:62642] Script timed out before returning headers: 
graphData.cgi, referer: https://ps-test.psc.edu/toolkit/
[Wed Jan 16 17:06:44.766389 2019] [cgi:warn] [pid 9366] [client 
128.182.160.119:62642] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
https://ps-test.psc.edu/toolkit/

...<snip>

[Fri Jan 18 15:40:08.607581 2019] [cgi:warn] [pid 9729] [client 
68.42.74.23:50797] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
https://192.231.244.54/toolkit/
[Fri Jan 18 15:40:08.607629 2019] [cgi:error] [pid 9729] [client 
68.42.74.23:50797] Script timed out before returning headers: graphData.cgi, 
referer: https://192.231.244.54/toolkit/
[Fri Jan 18 15:41:08.667106 2019] [cgi:warn] [pid 9729] [client 
68.42.74.23:50797] AH01220: Timeout waiting for output from CGI script 
/usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer: 
https://192.231.244.54/toolkit/

Kathy


On 1/24/2019 1:19 PM, Michael Johnson wrote:
> Hi Kathy,
>
> It does appear that your measurement archive is working; however, it 
> appears that the CGI that retrieves test result data to create the test 
> listing is timing out. i.e.
>
> http://192.231.244.54/perfsonar-graphs/cgi-bin/graphData.cgi?action=test_list&timeperiod=604800,86400&url=http%3A%2F%2F192.231.244.54%2Fesmond%2Fperfsonar%2Farchive%2F 
>
>
> Do you see some issues in /var/log/httpd/ssl_error_log relating to 
> graphData.cgi? This could shed some light on what's going on.
>
> Thanks,
> Michael
>
> On Thu, Jan 24, 2019 at 09:12:06AM -0800, Andrew Lake wrote:
> > Hi Kathy,
> >
> > Sorry, I realized I lost both user list and Michael from the CC. My
> > apologies, must have been going too fast. Both are added back now.
> >
> > Thanks,
> > Andy
> >
> >
> > On January 24, 2019 at 4:51:30 AM, Kathy Benninger 
> > ()
> > wrote:
> >
> > Hi Andy,
> >
> > I haven't heard from Michael and was wondering if he had recommendations on
> > how to debug the apparent time-out problem I'm seeing. His email address
> > didn't show up in your reply so I can't bug him directly  :-) .
> >
> > Thanks,
> > Kathy
> >
> >
> > On 1/18/2019 3:49 PM, Andrew Lake wrote:
> >
> > Hi Kathy,
> >
> > Thanks for the addresses, cassandra appears to be working fine. I can get
> > data out of your archive. Examples:
> >
> > http://192.231.244.54/esmond/perfsonar/archive/?time-range=86400
> > http://192.231.244.54/esmond/perfsonar/archive/013df91b72784f1ebfc16746cbfbf706/packet-loss-rate-bidir/base?time-range=86400 
> >
> >
> > For some reason the javascript is timing-put when creating that listing.
> > Michael, CC’ed may have some insight since he is our javascript expert.
> >
> > Thanks,
> > Andy
> >
> >
> >
> > On January 18, 2019 at 3:23:46 PM, Kathy Benninger 
> > ()
> > wrote:
> >
> > Hi Andy,
> >
> > I tried the restart, but still get the same error on the homepage:
> > Error loading test listing; measurement archive unreachable:
> > http://192.231.244.54/esmond/perfsonar/archive/
> >
> > It looks like cassandra is running after the restart:
> >
> > [root@ps-test log]# pkill -9 -f java
> > [root@ps-test log]# systemctl restart cassandra
> >
> >
> > [root@ps-test log]# systemctl status cassandra
> > ● cassandra.service - SYSV: Starts and stops Cassandra
> > Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
> > Active: active (exited) since Fri 2019-01-18 14:49:57 EST; 4s ago
> > Docs: man:systemd-sysv-generator(8)
> > Process: 10129 ExecStop=/etc/rc.d/init.d/cassandra stop (code=exited,
> > status=1/FAILURE)
> > Process: 10185 ExecStart=/etc/rc.d/init.d/cassandra start (code=exited,
> > status=0/SUCCESS)
> >
> > Jan 18 14:49:57 ps-test.psc.edu cassandra[10129]: Jan 18 14:49:52
> > ps-test.psc.edu su[10139]: (to cassandra) root on none
> > Jan 18 14:49:57 ps-test.psc.edu cassandra[10129]: Jan 18 14:49:52
> > ps-test.psc.edu cassandra[10129]: Shutdown Cassandra: bash: line 0: kill:
> > (6169) - No such process
> > Jan 18 14:49:57 ps-test.psc.edu systemd[1]: cassandra.service: control
> > process exited, code=exited status=1
> > Jan 18 14:49:57 ps-test.psc.edu systemd[1]: Stopped SYSV: Starts and stops
> > Cassandra.
> > Jan 18 14:49:57 ps-test.psc.edu systemd[1]: Unit cassandra.service entered
> > failed state.
> > Jan 18 14:49:57 ps-test.psc.edu systemd[1]: cassandra.service failed.
> > Jan 18 14:49:57 ps-test.psc.edu systemd[1]: Starting SYSV: Starts and stops
> > Cassandra...
> > Jan 18 14:49:57 ps-test.psc.edu su[10194]: (to cassandra) root on none
> > Jan 18 14:49:57 ps-test.psc.edu cassandra[10185]: Starting Cassandra: OK
> > Jan 18 14:49:57 ps-test.psc.edu systemd[1]: Started SYSV: Starts and stops
> > Cassandra.
> > [root@ps-test log]#
> >
> > Do you have some other ideas about what to try? The fact that the problem
> > persists after a Toolkit reinstall really has me baffled.
> >
> > Thanks,
> > Kathy
> >
> >
> > On 1/18/2019 2:11 PM, Andrew Lake wrote:
> > > Hi Kathy,
> > >
> > > Have you tried restarting cassandra? It’s not uncommon for it to get stuck
> > > or stop running with little indication unfortunately. Usually the
> > following
> > > is a pretty sure-fire way to make sure cassandra is stopped and then
> > restart it:
> > > pkill 9 -f java
> > > systemctl restart cassasandra
> > >
> > > Thanks,
> > > Andy
> > >
> > >
> > > On January 18, 2019 at 11:15:17 AM, Kathy Benninger 
> > > (
> > > <mailto:>)
> > >  wrote:
> > >
> > > > Attached are the cassandra/cassandra.log and cassandra/system.log files
> > as
> > > > it's likely I could be staring at the problem and not recognizing it.
> > > >
> > > > Thanks,
> > > > Kathy
> > > >
> > > >
> > > > On 1/17/2019 1:33 PM, Kathy Benninger wrote:
> > > > > Hi Szymon,
> > > > >
> > > > > cassandra/cassandra.log and cassandra/system.log look similar to log
> > files
> > > > > on other hosts in the mesh. There's nothing that jumps out as clearly
> > > > > indicating a problem
> > > > >
> > > > > There are 11 hosts in the mesh. Each runs both throughput and latency
> > tests
> > > > > on the same interface.
> > > > >
> > > > > Kathy
> > > > >
> > > > >
> > > > > On 1/17/2019 9:34 AM, Szymon Trocha wrote:
> > > > >> W dniu 17.01.2019 o 15:11, Kathy Benninger pisze:
> > > > >>> A perfSONAR built with NetInstall Toolkit v4.1.5-1.el7 from late
> > December
> > > > >>> is reporting the following error on its homepage under "Test
> > Results":
> > > > >>>
> > > > >>>   Error loading test listing; measurement archive unreachable:
> > > > >>> http://192.231.244.54/esmond/perfsonar/archive/
> > > > >>>
> > > > >>> The pS is running tests and responds to test requests from other pSs
> > in
> > > > >>> the same meshconfig file.
> > > > >>>
> > > > >>> httpd/error_log shows timeouts when I bring up the homepage, e.g.,:
> > > > >>>
> > > > >>> [Wed Jan 16 11:17:46.585696 2019] [cgi:warn] [pid 21553] [client
> > > > >>> 128.182.160.119:52235 <http://128.182.160.119:52235>] AH01220:
> > Timeout
> > > > waiting for output from CGI
> > > > >>> script /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer:
> > > > >>> http://192.231.244.54/toolkit/
> > > > >>> [Wed Jan 16 11:17:46.585765 2019] [cgi:error] [pid 21553] [client
> > > > >>> 128.182.160.119:52235 <http://128.182.160.119:52235>] Script timed
> > out
> > > > before returning headers:
> > > > >>> graphData.cgi, referer: http://192.231.244.54/toolkit/
> > > > >>> [Wed Jan 16 11:21:48.043369 2019] [cgi:warn] [pid 19343] [client
> > > > >>> 128.182.160.119:52338 <http://128.182.160.119:52338>] AH01220:
> > Timeout
> > > > waiting for output from CGI
> > > > >>> script /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer:
> > > > >>> http://192.231.244.54/toolkit/
> > > > >>> [Wed Jan 16 11:21:48.043438 2019] [cgi:error] [pid 19343] [client
> > > > >>> 128.182.160.119:52338 <http://128.182.160.119:52338>] Script timed
> > out
> > > > before returning headers:
> > > > >>> graphData.cgi, referer: http://192.231.244.54/toolkit/
> > > > >>> [Wed Jan 16 11:33:43.135517 2019] [cgi:warn] [pid 23720] [client
> > > > >>> 128.182.160.119:52536 <http://128.182.160.119:52536>] AH01220:
> > Timeout
> > > > waiting for output from CGI
> > > > >>> script /usr/lib/perfsonar/graphs/cgi-bin/graphData.cgi, referer:
> > > > >>> http://192.231.244.54/toolkit/
> > > > >>> [Wed Jan 16 11:33:43.135577 2019] [cgi:error] [pid 23720] [client
> > > > >>> 128.182.160.119:52536 <http://128.182.160.119:52536>] Script timed
> > out
> > > > before returning headers:
> > > > >>> graphData.cgi, referer: http://192.231.244.54/toolkit/
> > > > >>>
> > > > >>> esmond/django.log:
> > > > >>>
> > > > >>> 2019-01-16 10:32:05,652 [ERROR]
> > > > >>>
> > /usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py: 
> >
> > > > >>> Internal Server Error:
> > > > >>> /esmond/perfsonar/archive/7c9ba2a49ac24a42833efaeec83e6ab7/
> > > > >>> Traceback (most recent call last):
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py", 
> >
> > > > >>> line 42, in inner
> > > > >>>     response = get_response(request)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py",
> > > > >>> line 249, in _legacy_get_response
> > > > >>>     response = self._get_response(request)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py",
> > > > >>> line 187, in _get_response
> > > > >>>     response = self.process_exception_by_middleware(e, request)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py",
> > > > >>> line 185, in _get_response
> > > > >>>     response = wrapped_callback(request, *callback_args,
> > **callback_kwargs)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py", 
> >
> > > > >>> line 58, in wrapped_view
> > > > >>>     return view_func(*args, **kwargs)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py",
> > > > >>> line 90, in view
> > > > >>>     return self.dispatch(request, *args, **kwargs)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py",
> > > > >>> line 489, in dispatch
> > > > >>>     response = self.handle_exception(exc)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py",
> > > > >>> line 449, in handle_exception
> > > > >>>     self.raise_uncaught_exception(exc)
> > > > >>>   File
> > > > >>>
> > "/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py",
> > > > >>> line 486, in dispatch
> > > > >>>     response = handler(request, *args, **kwargs)
> > > > >>>   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 887,
> > in update
> > > > >>>     check_connection()
> > > > >>>   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 78, in
> > > > >>> check_connection
> > > > >>>     db = CASSANDRA_DB(get_config(get_config_path()))
> > > > >>>   File "/usr/lib/esmond/esmond/cassandra.py", line 134, in __init__
> > > > >>>     "at %s - %s" % (config.cassandra_servers[0], e))
> > > > >>> ConnectionException: "System Manager can't connect to Cassandra at
> > > > >>> localhost:9160 - Could not connect to localhost:9160"
> > > > >>>
> > > > >>> I've tried re-installing the Toolkit, but end up in the same
> > situation.
> > > > >>> Oddly, a new install works correctly for ~5 minutes (i.e., all the
> > tests
> > > > >>> are listed and linked on the homepage), but then it eventually starts
> > > > >>> returning the error that the archive is unreachable.
> > > > >>>
> > > > >>> Thoughts on how to debug or what could be wrong?
> > > > >>
> > > > >>
> > > > >> Hi Kathy,
> > > > >>
> > > > >> is there anything special in /var/log/cassandra/ ?
> > > > >>
> > > > >> Are there many tests to load in your host?
> > > > >>
> > > > >> Regards,
> > > > >>
> > > > >> --
> > > > >> Szymon Trocha
> > > > >> Poznań Supercomputing & Networking Center
> > > > >> General NOC phone +48 61-858-2015 | noc.pcss.pl <http://noc.pcss.pl> <
> > http://noc.pcss.pl>
> > > > >> Personal desk phone +48 61-858-2022
> > > > >> Wysłaliśmy do Ciebie ten e-mail w odpowiedzi na Twoje zapytanie lub w
> > > > >> związku z oferowaną usługą. Przesłanie korespondencji do Centrum
> > > > >> Zarządzania PCSS lub zgłoszenie telefoniczne jest równoznaczne z
> > > > >> wyrażeniem zgody na przetwarzanie danych osobowych przez Instytut
> > Chemii
> > > > >> Bioorganicznej Polskiej Akademii Nauk w Poznaniu adres: ul. Z.
> > > > >> Noskowskiego 12/14, 61-704 Poznań. Szczegółowe informacje znajdują
> > się w
> > > > >> naszej Polityce prywatności
> > > > >> <http://noc.pcss.pl/index.html#PolitykaPrywatnosci>. | This message
> > has
> > > > >> been sent as a part of communication with PSNC NOC or your service
> > request
> > > > >> sent to us. For more information read our Privacy Policy
> > > > >> <http://noc.psnc.pl/index.html#PrivacyPolicy>.
> > > > >>
> > > > --
> > > > To unsubscribe from this list:
> > > > https://lists.internet2.edu/sympa/signoff/perfsonar-user