Details
-
Bug
-
Status: Resolved
-
Urgent
-
Resolution: Duplicate
-
None
-
Critical
Description
The steps are as follows.
we have 4/4 node cassandra running in 3.9 version.
In one node made some changes to cassanra.yaml. issued a nodetool drain
killed the cassandra process and restarted the node. After sometime nodetool status reported multiple nodes are down in that DC.
Went and check the system.log of all the files and found the hint corruption occuring(CASSANDRA-12728). nodetool drain causing this corruption and bringing multiple nodes down is a big concern.
ERROR [HintsDispatcher:2] 2016-10-26 12:17:59,361 HintsDispatchExecutor.java:225 - Failed to dispatch hints file 4d1362f0-053c-4042-80a7-bfc85a26c90f-1477509190999-1.hints: file is corrupted ({})
org.apache.cassandra.io.FSReadError: java.io.EOFException
at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259) [apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242) [apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220) [apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199) [apache-cassandra-3.9.jar:3.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]
Attachments
Issue Links
- duplicates
-
CASSANDRA-12728 Handling partially written hint files
- Resolved
- relates to
-
CASSANDRA-12728 Handling partially written hint files
- Resolved