Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
Normal
Description
I recently, and probably mistakenly, upgraded one of my production C* clusters to 2.2.0. I am seeing these errors in the logs, followed by an intense period of garbage collection until the node, then the ring, becomes crippled:
ERROR [OptionalTasks:1] 2015-08-04 03:24:56,057 CassandraDaemon.java:182 - Exception in thread Thread[OptionalTasks:1,5,main] java.lang.IllegalStateException: Unable to compute when histogram overflowed at org.apache.cassandra.utils.EstimatedHistogram.percentile(EstimatedHistogram.java:179) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getValue(EstimatedHistogramReservoir.java:84) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.db.ColumnFamilyStore$3.run(ColumnFamilyStore.java:405) ~[apache-cassandra-2.2.0.jar:2.2.0] at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.2.0.jar:2.2.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
I am not sure if the GC instability is this or something else, but I though this histogram overflow issue was fixed in 2.1.3? Anyway, reporting now as a possible regression. Please let me know what I can provide in terms of information to help with this. Thanks!