Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7109

Indexing threads stuck during network partition can put leader into down state

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.10.3, 5.0
    • 5.1, 6.0
    • SolrCloud
    • None

    Description

      I found this recently while running some Jepsen tests. I found that some threads get stuck on zk operations for a long time in ZkController.updateLeaderInitiatedRecoveryState method and when they wake up they go ahead with setting the LIR state to down. But in the mean time, new leader has been elected and sometimes you'd get into a state where the leader itself is put into recovery causing the shard to reject all writes.

      Attachments

        1. SOLR-7109.patch
          14 kB
          Shalin Shekhar Mangar
        2. SOLR-7109.patch
          17 kB
          Shalin Shekhar Mangar

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-8069 Ensure that only the valid ZooKeeper registered leader can put a replica into Leader Initiated Recovery.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-7245 Temporary ZK election or connection loss should not stall indexing due to LIR

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              shalin Shalin Shekhar Mangar
              shalin Shalin Shekhar Mangar
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: