Details
-
Bug
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
4.2.0
-
None
-
None
Description
Clone from CURATOR-504.
We use LeaderLatch in a lot of places in our system and when ZooKeeper ensemble is unstable and clients are reconnecting to logs are full of messages like the following:
[2017-08-31 19:18:34,562][ERROR][org.apache.curator.framework.recipes.leader.LeaderLatch] Can't find our node. Resetting. Index: -1 {}
According to the implementation, this can happen in two cases:
- When internal state `ourPath` is null
- When the list of latches does not have the expected one.
I believe we hit the first condition because of races that occur after client reconnects to ZooKeeper.
- Client reconnects to ZooKeeper and LeaderLatch gets the event and calls reset method which set the internal state (`ourPath`) to null, removes old latch and creates a new one. This happens in thread "Curator-ConnectionStateManager-0".
- Almost simultaneously, LeaderLatch gets another even NodeDeleted (here) and tries to re-read the list of latches and check leadership. This happens in the thread "main-EventThread".
Therefore, sometimes there is a situation when method `checkLeadership` is called when `ourPath` is null.
Attachments
Issue Links
- is a clone of
-
CURATOR-504 Race conditions in LeaderLatch after reconnecting to ensemble
- Resolved
- is related to
-
CURATOR-505 A circuit breaking ConnectionStateListener would be very helpful
- Resolved
- relates to
-
FLINK-29173 Upgrade curator
- Closed