Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-26383

HBCK incorrectly reports inconsistencies for recently split regions following a master failover

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.4.3
    • 2.5.0, 3.0.0-alpha-2, 2.4.8
    • master
    • None
    • Reviewed

    Description

      When a region P splits into A and B, following a master failover the newly active master reports that P is in an inconsistent state. This seems to be a regression introduced in HBASE-25847 (cc andrew.purtell@gmail.com) which changed regionInfo.isParentSplit() to regionState.isSplit(). The region state after restart is CLOSED (rather than SPLIT), so both region state and region info should be checked, presumably with regionState.isSplit() || regionInfo.isSplit(). This situation resolves itself on its own when a major compaction occurs and P is GCed, but having the master incorrectly report inconsistencies is pretty bad. We had a pretty big outage due to a series of operator errors as our SRE team was trying to fix this inconsistency that, in fact, didn't even exist.

      Thanks to Stack for helping look over this issue and Vlad Hanciuta for root causing the bug.

      Attachments

        Issue Links

          Activity

            People

              apurtell Andrew Kyle Purtell
              tsuna Benoit Sigoure
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: