perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

FYI, I'm not sure how this happened, but the ownership of a cassandra directory broke on one of the nodes I was working on.

Symptom: psGraph.cgi was giving an infinite spinner.

After rebooting the box, instead I saw:
Error retrieving test data: undefined

Apache error log showed:
...
[Wed Sep 09 23:52:55 2015] [error] [client 127.0.0.1]   File "/opt/esmond/esmond/api/perfsonar/api.py", line 41, in <module>
[Wed Sep 09 23:52:55 2015] [error] [client 127.0.0.1]     raise ConnectionException(error_msg)
[Wed Sep 09 23:52:55 2015] [error] [client 127.0.0.1] ConnectionException: 'Unable to connect to cassandra. Please verify cassandra is running.'

So I did /etc/init.d/cassandra start, but still no daemon was running. Checked cassandra logs:

[root@pfsnr ~]# tail /var/log/cassandra/cassandra.log
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2155)
    at org.apache.cassandra.db.ColumnFamilyStore.truncateBlocking(ColumnFamilyStore.java:2103)
    at org.apache.cassandra.db.SystemKeyspace.discardCompactionsInProgress(SystemKeyspace.java:241)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:656)
Caused by: java.io.IOException: Unable to create directory /var/lib/cassandra/data/system/compactions_in_progress/snapshots/1441832009319-compactions_in_progress
    ... 12 more
Exception encountered during startup: java.io.IOException: Unable to create directory /var/lib/cassandra/data/system/compactions_in_progress/snapshots/1441832009319-compactions_in_progress

Checking permissions:

[root@pfsnr ~]# ls -ld /var/lib/cassandra/data/system/compactions_in_progress/snapshots/
drwxr-xr-x 3 root root 4096 Sep  9 12:46 /var/lib/cassandra/data/system/compactions_in_progress/snapshots/
[root@pfsnr ~]# chown cassandra:cassandra /var/lib/cassandra/data/system/compactions_in_progress/snapshots/
[root@pfsnr ~]# /etc/init.d/cassandra start
Starting Cassandra: OK

(on a different system I found this directory owned by cassandra:cassandra)

And after this it was fine. But I wonder how it ended up being owned by root:root?

Also, it would be nice if a more informative error were returned to the client if cassandra is not running.

Regards,

Brian.