Uploaded image for project: 'Apache Curator'
  1. Apache Curator
  2. CURATOR-644

CLONE - Race conditions in LeaderLatch after reconnecting to ensemble

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 4.2.0
    • 5.4.0
    • None
    • None

    Description

      Clone from CURATOR-504.

      We use LeaderLatch in a lot of places in our system and when ZooKeeper ensemble is unstable and clients are reconnecting to logs are full of messages like the following:

      [2017-08-31 19:18:34,562][ERROR][org.apache.curator.framework.recipes.leader.LeaderLatch] Can't find our node. Resetting. Index: -1 {}

      According to the implementation, this can happen in two cases:

      • When internal state `ourPath` is null
      • When the list of latches does not have the expected one.

      I believe we hit the first condition because of races that occur after client reconnects to ZooKeeper.

      • Client reconnects to ZooKeeper and LeaderLatch gets the event and calls reset method which set the internal state (`ourPath`) to null, removes old latch and creates a new one. This happens in thread "Curator-ConnectionStateManager-0".
      • Almost simultaneously, LeaderLatch gets another even NodeDeleted (here) and tries to re-read the list of latches and check leadership. This happens in the thread "main-EventThread".

      Therefore, sometimes there is a situation when method `checkLeadership` is called when `ourPath` is null.

      Attachments

        Issue Links

          Activity

            People

              tison Zili Chen
              rikimberley Ken Huang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8.5h
                  8.5h