Description
We have discovered issues with corrupted SSTables.
ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - Cannot read sstable /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db, Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, Index.db, Filter.db]; other IO error, skipping table java.io.EOFException: EOF after 1898 bytes out of 21093 at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561) ~[apache-cassandra-3.11.0.jar:3.11.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.0.jar:3.11.0]
Files look like this:
-rw-r--r--. 1 cassandra cassandra 3899251 Aug 7 08:37 mc-6166-big-CompressionInfo.db -rw-r--r--. 1 cassandra cassandra 16874421686 Aug 7 08:37 mc-6166-big-Data.db -rw-r--r--. 1 cassandra cassandra 10 Aug 7 08:37 mc-6166-big-Digest.crc32 -rw-r--r--. 1 cassandra cassandra 2930904 Aug 7 08:37 mc-6166-big-Filter.db -rw-r--r--. 1 cassandra cassandra 111175880 Aug 7 08:37 mc-6166-big-Index.db -rw-r--r--. 1 cassandra cassandra 13762 Aug 7 08:37 mc-6166-big-Statistics.db -rw-r--r--. 1 cassandra cassandra 882008 Aug 7 08:37 mc-6166-big-Summary.db -rw-r--r--. 1 cassandra cassandra 92 Aug 7 08:37 mc-6166-big-TOC.txt
Attachments
Issue Links
- duplicates
-
CASSANDRA-13756 StreamingHistogram is not thread safe
- Resolved
- is broken by
-
CASSANDRA-6916 Preemptive opening of compaction result
- Resolved
- is related to
-
CASSANDRA-13038 33% of compaction time spent in StreamingHistogram.update()
- Resolved
- relates to
-
CASSANDRA-13756 StreamingHistogram is not thread safe
- Resolved