Details
-
Improvement
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
4.0.10, 4.1.2, 5.0-alpha1, 5.0
-
None
-
Operability
-
Low Hanging Fruit
-
All
-
None
-
Description
When compaction fails, the log message should list a) the free space available on disk at that point in time and b) perhaps the number and/or size of the source sstables being compacted.
Free space can change from one moment to the next, so when the below compaction failed because it needed 161GB, upon checking the server a few minutes later, it had 184GB free. Similarly, the error message mentions it was writing out one SSTable on this STCS table, but its not clear if it was combining X -> 1 tables, or something else.
[INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1 CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1 sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s, Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375. Partition merge counts were \{1:21375, }
[ERROR] [CompactionExecutor:77774] cluster_id=87 ip_address=127.1.1.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:77774,1,main]
java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 161228934093
at org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
at org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613)
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
