perfSONAR User Q&A and Other Discussion

[Johann Hugo <>]

[root@perf-cm ~]# systemctl status cassandra
● cassandra.service - SYSV: Starts and stops Cassandra
   Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2023-05-10 14:48:39 SAST; 1h 6min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1163 ExecStart=/etc/rc.d/init.d/cassandra start (code=exited, status=0/SUCCESS)

May 10 14:48:37 perf-cm.sanren.ac.za systemd[1]: Starting SYSV: Starts and stops Cassandra...
May 10 14:48:38 perf-cm.sanren.ac.za su[1204]: (to cassandra) root on none
May 10 14:48:39 perf-cm.sanren.ac.za cassandra[1163]: Starting Cassandra: OK
May 10 14:48:39 perf-cm.sanren.ac.za systemd[1]: Started SYSV: Starts and stops Cassandra.

[root@perf-cm ~]# systemctl stop cassandra

[root@perf-cm ~]# systemctl start cassandra

[root@perf-cm ~]# tail -F /var/log/messages | grep cassandra
May 10 15:55:05 perf-cm su: (to cassandra) root on none
May 10 15:55:05 perf-cm systemd: Created slice User Slice of cassandra.
May 10 15:55:05 perf-cm systemd: Started Session c2 of user cassandra.
May 10 15:55:05 perf-cm cassandra: Shutdown Cassandra: bash: line 0: kill: (1640) - No such process
May 10 15:55:05 perf-cm systemd: Removed slice User Slice of cassandra.
May 10 15:55:10 perf-cm cassandra: ERROR: could not stop cassandra:  ● cassandra.service - SYSV: Starts and stops Cassandra
May 10 15:55:10 perf-cm cassandra: Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
May 10 15:55:10 perf-cm cassandra: Active: deactivating (stop) since Wed 2023-05-10 15:55:05 SAST; 5s ago
May 10 15:55:10 perf-cm cassandra: Docs: man:systemd-sysv-generator(8)
May 10 15:55:10 perf-cm cassandra: Process: 1163 ExecStart=/etc/rc.d/init.d/cassandra start (code=exited, status=0/SUCCESS)
May 10 15:55:10 perf-cm cassandra: Control: 26634 (cassandra)
May 10 15:55:10 perf-cm cassandra: CGroup: /system.slice/cassandra.service
May 10 15:55:10 perf-cm cassandra: └─control
May 10 15:55:10 perf-cm cassandra: ├─26634 /bin/bash /etc/rc.d/init.d/cassandra stop
May 10 15:55:10 perf-cm cassandra: ├─26686 /bin/bash /etc/rc.d/init.d/cassandra status
May 10 15:55:10 perf-cm cassandra: └─26689 systemctl status cassandra.service
May 10 15:55:10 perf-cm cassandra: May 10 14:48:37 perf-cm.sanren.ac.za systemd[1]: Starting SYSV: Starts and stops Cassandra...
May 10 15:55:10 perf-cm cassandra: May 10 14:48:38 perf-cm.sanren.ac.za su[1204]: (to cassandra) root on none
May 10 15:55:10 perf-cm cassandra: May 10 14:48:39 perf-cm.sanren.ac.za cassandra[1163]: Starting Cassandra: OK
May 10 15:55:10 perf-cm cassandra: May 10 14:48:39 perf-cm.sanren.ac.za systemd[1]: Started SYSV: Starts and stops Cassandra.
May 10 15:55:10 perf-cm cassandra: May 10 15:55:05 perf-cm.sanren.ac.za systemd[1]: Stopping SYSV: Starts and stops Cassandra...
May 10 15:55:10 perf-cm cassandra: May 10 15:55:05 perf-cm.sanren.ac.za su[26644]: (to cassandra) root on none
May 10 15:55:10 perf-cm cassandra: May 10 15:55:05 perf-cm.sanren.ac.za cassandra[26634]: Shutdown Cassandra: bash: line 0: kill: (1640) - No such process
May 10 15:55:10 perf-cm systemd: cassandra.service: control process exited, code=exited status=1
May 10 15:55:10 perf-cm systemd: Unit cassandra.service entered failed state.
May 10 15:55:10 perf-cm systemd: cassandra.service failed.


May 10 15:55:28 perf-cm su: (to cassandra) root on none
May 10 15:55:28 perf-cm systemd: Created slice User Slice of cassandra.
May 10 15:55:28 perf-cm systemd: Started Session c3 of user cassandra.
May 10 15:55:28 perf-cm cassandra: Starting Cassandra: OK
May 10 15:55:32 perf-cm systemd: Removed slice User Slice of cassandra.

On Wed, May 10, 2023 at 3:16 PM Szymon Trocha <> wrote:

Hi Johann,

W dniu 10.05.2023 o 14:59, Johann Hugo pisze:

Cassandra looks fine

tail -F /var/log/cassandra/cassandra.log

        at org.apache.cassandra.io.sstable.SSTableReader.openMetadata(SSTableReader.java:241)
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:204)
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:193)
        at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:280)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
 INFO 14:43:55,826 Opening /var/lib/cassandra/data/esmond/raw_data/esmond-raw_data-jb-65184 (168418794 bytes)

I'm not sure Cassandra looks fine. The above is the end of longer java exception.

what does "systemctl status cassandra" say ?

Esmond is running

[root@perf-cm ~]# tail -F /var/log/esmond/esmond.log
2023-05-03 00:30:07,079 [INFO] /usr/lib/esmond/esmond/cassandra.py: Checking/creating column families
2023-05-03 00:30:07,081 [INFO] /usr/lib/esmond/esmond/cassandra.py: Schema check done
2023-05-03 00:30:07,094 [INFO] /usr/lib/esmond/esmond/cassandra.py: Connected to ['localhost:9160']

Django is not happy

tail -F /var/log/esmond/django.log
  Traceback (most recent call last):

  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/esmond/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/viewsets.py", line 116, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/esmond/lib/python3.6/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 921, in update
    check_connection()
  File "/usr/lib/esmond/esmond/api/perfsonar/api_v2.py", line 75, in check_connection
    db = CASSANDRA_DB(get_config(get_config_path()))
  File "/usr/lib/esmond/esmond/cassandra.py", line 135, in __init__
    "at %s - %s" % (config.cassandra_servers[0], e))
esmond.cassandra.ConnectionException: "System Manager can't connect to Cassandra at localhost:9160 - Could not connect to any of [('::1', 9160, 0, 0), ('127.0.0.1', 9160)]"

it can't connect because Cassandra is probably not running properly

Regards,

--

Szymon Trocha
Poznań Supercomputing & Networking Center

General NOC phone +48 61-858-2015 | noc.pcss.pl
Personal desk phone +48 61-858-2022

Wysłaliśmy do Ciebie ten e-mail w odpowiedzi na Twoje zapytanie lub w związku z oferowaną usługą. Przesłanie korespondencji do Centrum Zarządzania PCSS lub zgłoszenie telefoniczne jest równoznaczne z wyrażeniem zgody na przetwarzanie danych osobowych przez Instytut Chemii Bioorganicznej Polskiej Akademii Nauk w Poznaniu adres: ul. Z. Noskowskiego 12/14, 61-704 Poznań. Szczegółowe informacje znajdują się w naszej Polityce prywatności. | This message has been sent as a part of communication with PSNC NOC or your service request sent to us. For more information read our Privacy Policy.