Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
Correctness - API / Semantic Implementation
-
Normal
-
Normal
-
User Report
-
All
-
None
-
Description
In one of our test environments we encountered the following exception:
2020-02-02T10:50:13.276+0100 [GossipTasks:1] ERROR o.a.c.u.NoSpamLogger$NoSpamLogStatement:97 log java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1 at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178) at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465) at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895) at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78) at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1 at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178) [apache-cassandra-3.11.5.jar:3.11.5] at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465) [apache-cassandra-3.11.5.jar:3.11.5] at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895) [apache-cassandra-3.11.5.jar:3.11.5] at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78) [apache-cassandra-3.11.5.jar:3.11.5] at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240) [apache-cassandra-3.11.5.jar:3.11.5] at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) [apache-cassandra-3.11.5.jar:3.11.5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_231] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_231] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_231] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_231] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_231] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_231] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.42.Final.jar:4.1.42.Final] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231]
Since CASSANDRA-15059 we check that all state changes are performed in the GossipStage but it seems like it was still performed in the "current" thread here. It should be as simple as adding a
runInGossipStageBlocking(() ->)
for it.
I'll upload patches for 3.0, 3.11 and 4.0.