perfSONAR User Q&A and Other Discussion

[Andrew Lake <>]

Hi,

Maybe you did this, but you might want to make sure there is no old cassandra process stuck in the runnings before trying the repair steps. If you do a ‘ps auxw | grep java’ you should not get anything back. If there is anything, you should kill it. Something like 'pkill -9 -f java' will do the trick. After that you should hopefully be able to complete the steps.

Thanks,

Andy

On July 6, 2017 at 2:51:19 PM, Tao Zhang () wrote:

Hi perfSONAR community,

 

I encountered a database issue that I don’t know how to fix.

 

In the perfSONAR GUI, the test results can not be retrieved. (Error loading test listing: Internal Server Error).

 

I check the Esmond log:

 

2017-07-06 14:36:27,408 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,411 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,444 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,447 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,448 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,448 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,450 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,451 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,491 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:27,493 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

2017-07-06 14:36:30,898 [ERROR] /usr/lib/esmond/esmond/api/perfsonar/api_v2.py: Unable to connect to cassandra. Please verify cassandra is running.

 

The log show it is Cassandra issue, so I check Cassandra log:

 

ERROR 14:31:25,549 Exception encountered during startup

org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/var/lib/cassandra/data/system/local/system-local-jb-88-Data.db): corruption detected, chunk at 0 of length 142.

                at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:91)

                at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:283)

                at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)

                at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1284)

                at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:57)

                at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)

                at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)

                at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:238)

                at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)

                at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)

                at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)

                at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1619)

                at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1438)

                at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:340)

                at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:89)

                at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:318)

                at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:338)

                at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:67)

                at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)

                at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:559)

                at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:320)

                at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:554)

                at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:643)

Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/var/lib/cassandra/data/system/local/system-local-jb-88-Data.db): corruption detected, chunk at 0 of length 142.

                at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:122)

                at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:87)

                ... 22 more

Caused by: java.io.IOException: Compressed lengths mismatch

                at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:98)

                at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:118)

                ... 23 more

 

The Cassandra log indicates that there is a CorruptSSTableException issue. I tried to use the nodetool to fix the problem.

 

[root@node1 conf]# nodetool status

Failed to connect to '127.0.0.1:7199': Connection refused (Connection refused)

 

I googled some solutions, but none of them worked.

https://engineering.gosquared.com/dealing-corrupt-sstable-cassandra

https://www.liquidweb.com/kb/error-failed-to-connect-to-127-0-0-17199-connection-refused-cassandra-solved/

 

Does anyone has any idea or suggestions?

 

Thanks in advance

 

Tao