Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Cannot Reproduce
-
None
-
None
-
Ubuntu 12.04 | C* 2.1.2 | ruby-driver 1.2
-
Normal
Description
During a ruby driver endurance/duration test, the following error occurred:
/mnt/systemlogs/system.log:ERROR [main] 2015-01-17 21:18:25,780 CassandraDaemon.java:482 - Exception encountered during startup /mnt/systemlogs/system.log-java.lang.NullPointerException: null /mnt/systemlogs/system.log- at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:573) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:249) [main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465) [main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554) [main/:na]
Here is the system.log leading up to the error:
INFO [main] 2015-01-17 21:18:24,581 ColumnFamilyStore.java:278 - Initializing system.peers INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,593 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-169 (10533 bytes) INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,597 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-171 (10572 bytes) INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,598 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-170 (10581 bytes) INFO [main] 2015-01-17 21:18:24,609 ColumnFamilyStore.java:278 - Initializing system.local INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,613 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-679 (5257 bytes) INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,616 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-678 (5679 bytes)
Cassandra attempted to restart twice unsuccessfully (this error occurred twice) and then gave up; it seems like a corrupt Data.db file? The endurance test consists of a chaos rhino which randomly rolling restarts a node.
The only other significant error is TombstoneOverwhelmingException and is probably unrelated:
/mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,378 SliceQueryFilter.java:218 - Scanned over 100000 tombstones in system.hints; query aborted (see tombstone_failure_threshold) /mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,416 CassandraDaemon.java:170 - Exception in thread Thread[HintedHandoff:2,1,main] /mnt/systemlogs/system.log-org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null /mnt/systemlogs/system.log- at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:220) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:320) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1885) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1693) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:378) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:337) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88) ~[main/:na] /mnt/systemlogs/system.log- at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:548) ~[main/:na] /mnt/systemlogs/system.log- at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_25] /mnt/systemlogs/system.log- at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.7.0_25] /mnt/systemlogs/system.log- at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_25] /mnt/systemlogs/system.log- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_25] /mnt/systemlogs/system.log- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) ~[na:1.7.0_25] /mnt/systemlogs/system.log- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25] /mnt/systemlogs/system.log- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25] /mnt/systemlogs/system.log- at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Attachments
Issue Links
- is related to
-
-
- Resolved
-