perfSONAR User Q&A and Other Discussion

[Brian Candler <>]

On 14/12/2018 09:16, Brian Candler wrote:

Now the problem is clear: cassandra database is corrupt, and it fails to start.  How to fix this, short of reinstalling the whole node, is an issue which can be dealt with separately.

FYI, I found a this guide as a starting point.

# systemctl stop cassandra
# nodetool scrub
Failed to connect to '127.0.0.1:7199': Connection refused (Connection refused)
# sstablescrub
Missing arguments
usage: sstablescrub [options] <keyspace> <column_family>
...

And I learned a bit about where cassandra stores keyspaces and column families.  So my first attempt was to clean the esmond databases:

# ls /var/lib/cassandra/data/
esmond  system  system_traces
# ls /var/lib/cassandra/data/esmond/
base_rates  rate_aggregations  raw_data  stat_aggregations[root@perfsonar-border cassandra]
# sstablescrub esmond base_rates
... etc

These were fine - nothing needed fixing, but cassandra still did not start.

Looking at the log message more carefully, it's "system local" which is corrupt.

When I try to scrub this, it goes into an infinite loop:

# sstablescrub system local 2>&1 | head -50
ERROR 09:32:47,962 Unable to initialize MemoryMeter (jamm not specified as javaagent).  This means Cassandra will be unable to measure object sizes accurately and may consequently OOM.
Pre-scrub sstables snapshotted into snapshot pre-scrub-1544779968622
Scrubbing SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-41-Data.db') (11956 bytes)
Scrub of SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-41-Data.db') complete: 1 rows in new sstable and 0 empty (tombstoned) rows dropped
Scrubbing SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-42-Data.db') (12600 bytes)
Scrub of SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-42-Data.db') complete: 1 rows in new sstable and 0 empty (tombstoned) rows dropped
Scrubbing SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-jb-40-Data.db') (156 bytes)
WARNING: Error reading row (stacktrace follows):
Retrying from row index; data is 149 bytes starting at 7
WARNING: Retry failed too. Skipping to next row (retry's stacktrace follows)
WARNING: Error reading row (stacktrace follows):
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: Error reading row (stacktrace follows):
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: Error reading row (stacktrace follows):
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: Error reading row (stacktrace follows):
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: Error reading row (stacktrace follows):
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: Error reading row (stacktrace follows):
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: Error reading row (stacktrace follows):
WARNING: Row starting at position 7 is unreadable; skipping to next
WARNING: Error reading row (stacktrace follows):
... etc

I get the same with "-s" or "-n" flags.

Since I don't have any replicas to recover data from, I'm a bit stuck now.

Can I just uninstall and reinstall the cassandra package - or will this leave me with missing databases which won't be recreated automatically?  In which case, is a full node reinstall required here?

Thanks,

Brian.