perfSONAR User Q&A and Other Discussion

["Robinson, John-Paul" <>]

A follow up:

While cassandra does have a confusing status, it's log file suggests it 
is reasonably happy.  Looking at the django log however, indicates there 
is some issue, possibly in connecting to cassandra:

2018-09-21 16:43:57,522 [ERROR] 
/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py:
 
Internal Server Error: 
/esmond/perfsonar/archive/7137bcc08ef4407bb6018bef73a29e3c/
Traceback (most recent call last):
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/exception.py",
 
line 42, in inner
     response = get_response(request)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 249, in _legacy_get_response
     response = self._get_response(request)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 187, in _get_response
     response = self.process_exception_by_middleware(e, request)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/core/handlers/base.py", 
line 185, in _get_response
     response = wrapped_callback(request, *callback_args, **callback_kwargs)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/django/views/decorators/csrf.py",
 
line 58, in wrapped_view
     return view_func(*args, **kwargs)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/viewsets.py", 
line 90, in view
     return self.dispatch(request, *args, **kwargs)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", 
line 489, in dispatch
     response = self.handle_exception(exc)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", 
line 449, in handle_exception
     self.raise_uncaught_exception(exc)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/rest_framework/views.py", 
line 486, in dispatch
     response = handler(request, *args, **kwargs)
   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 932, in 
update
     obj.save()
   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 407, in save
     existing = PSTimeSeriesObject.query_database(self.metadata_key, 
self.event_type, 'base', None, int(self.time), int(self.time), 1)
   File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 473, in 
query_database
     cf='average', ts_min=begin_millis, ts_max=end_millis, 
column_count=max_results)
   File "/usr/lib/esmond/esmond/cassandra.py", line 650, in 
query_aggregation_timerange
     column_count=cols)
   File 
"/usr/lib/esmond/lib/python2.7/site-packages/pycassa/columnfamily.py", 
line 772, in multiget
     packed_keys[offset:offset + buffer_size], cp, sp, consistency)
   File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", 
line 576, in execute
     conn = self.get()
   File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", 
line 539, in get
     conn = self._new_if_required(self._pool_size)
   File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", 
line 520, in _new_if_required
     return self._create_connection()
   File "/usr/lib/esmond/lib/python2.7/site-packages/pycassa/pool.py", 
line 431, in _create_connection
     (exc.__class__.__name__, exc))
AllServersUnavailable: An attempt was made to connect to each of the 
servers twice, but none of the attempts succeeded. The last failure was 
TTransportException: Could not connect to localhost:9160


Interestingly I can connect to localhost:9160 via the csql client.

I'm looking to see if this django error was transient or is not.

John-Paul

On 09/21/2018 01:12 PM, Robinson, John-Paul wrote:
> Hi,
>
> We've been working on a setting up a test mesh for a workshop at SoX
> (all nodes on same lan, all 1GigE nics).
>
> Followed the steps as outlined here with minor adjustment for the
> maddash service name change in 4.1.x
>
> https://bozeman-fiona-workshop.ucsd.edu/materials/exercises-perfsonar-centralmanagement.htm
>
>
> The mesh config was built from the skeleton example and published by
> psconfig to:
>
> https://soxws14.sox.net/psconfig/mesh-soxws.json
>
> The nodes in the mesh seem to be picking up the config and the maddash
> is displaying the grid:
>
> https://soxws14.sox.net/maddash-webui/index.cgi?dashboard=SoX%20FIONA%20Workshop%20Measurements
>
> There appears to be a problem ingesting the test results into the esmond
> though.  The configuration has been up for several hours but only tests
> originating from the host running the maddash are appearing in the data.
>
> We populated the permissions for the esmond like so, where the input
> provides the IP of each node in the mesh:
>
> for node in `cat soxws-nodes`
> do
>     ip=`host $node | cut -d" " -f4`
>     echo $ip
>     sudo /usr/sbin/esmond_manage add_user_ip_address example_user  $ip
> 143.215.191.0/26
> done
>
> We suspect something is up with cassandra the status shows somewhat
> confusing output "active (exited)":
>
> workshop@soxws14:~/projects/mesh$ sudo systemctl status cassandra
> ● cassandra.service - SYSV: Starts and stops Cassandra
>      Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset:
> disabled)
>      Active: active (exited) since Fri 2018-09-21 16:43:46 UTC; 24min ago
>        Docs: man:systemd-sysv-generator(8)
>     Process: 5622 ExecStop=/etc/rc.d/init.d/cassandra stop (code=exited,
> status=1/FAILURE)
>     Process: 5934 ExecStart=/etc/rc.d/init.d/cassandra start
> (code=exited, status=0/SUCCESS)
>
> Sep 21 16:43:46 soxws14.sox.net systemd[1]: Starting SYSV: Starts and
> stops Cassandra...
> Sep 21 16:43:46 soxws14.sox.net su[5943]: (to cassandra) root on none
> Sep 21 16:43:46 soxws14.sox.net cassandra[5934]: Starting Cassandra: OK
> Sep 21 16:43:46 soxws14.sox.net systemd[1]: Started SYSV: Starts and
> stops Cassandra.
>
> There are reports of this behavior being associated with java versions
> and cassandra.  Having followed stock steps for perfsonar central
> management install, I wouldn't expect that to be an issue.
>
> Curious if there are any insights on potential issues or where to look.
>
> Thanks,
>
> John-Paul