Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-5359

AssertionError in HintedHandoff and CompactionExecutor

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Won't Fix
    • None
    • None
    • None

    • Centos 6.4
      3 Nodes
      RF 3
      Vnodes
      Murmur3Partitioner

    • Normal

    Description

      Seeing this on 2 of the 3 nodes pretty consistently. The stack traces seem to happen consistently together, and I believe they are related.

      HintedHandoff

      ERROR [HintedHandoff:3] 2013-03-18 20:58:58,643 CassandraDaemon.java (line 132) Exception in thread Thread[HintedHandoff:3,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 4
53516860 but now it is 453517031
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252)
        at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89)
        at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 453516860 but now
it is 453517031
        at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402)
        ... 7 more
Caused by: java.lang.AssertionError: originally calculated column size of 453516860 but now it is 453517031
        at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        ... 3 more
ERROR [CompactionExecutor:8] 2013-03-18 20:58:58,645 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:8,
1,main]
java.lang.AssertionError: originally calculated column size of 453516860 but now it is 453517031
        at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

      CompactionExecutor

       INFO [CompactionExecutor:9] 2013-03-18 21:02:54,486 CompactionController.java (line 158) Compacting large row system/hints:bcc4b89f-
a2ac-4abe-8420-4f5c2462a65b (453517091 bytes) incrementally
ERROR [CompactionExecutor:9] 2013-03-18 21:03:16,034 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:9,
1,main]
java.lang.AssertionError: originally calculated column size of 453516860 but now it is 453517031
        at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
ERROR [HintedHandoff:1] 2013-03-18 21:03:16,035 CassandraDaemon.java (line 132) Exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 4
53516860 but now it is 453517031
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252)
        at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89)
        at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 453516860 but now
it is 453517031
        at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402)
        ... 7 more
Caused by: java.lang.AssertionError: originally calculated column size of 453516860 but now it is 453517031
        at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        ... 3 more

      Please let me know any additional info needed.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-5720 AssertionError in CompactionExecutor

          • Normal -
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-4206 AssertionError: originally calculated column size of 629444349 but now it is 588008950

          • Normal -
          • Resolved

          Activity

            People

              Unassigned Unassigned
              joeyi Joey Imbasciano
              Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: