Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20828 Finish-up AMv2 Design/List of Tenets/Specification of operation
  3. HBASE-20975

Lock may not be taken or released while rolling back procedure

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0, 2.0.1
    • 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.2
    • amv2
    • None

    Description

      Find this one when investigating HBASE-20921, too.

      Here is some code from executeRollback in ProcedureExecutor.java.

          boolean reuseLock = false;
          while (stackTail --> 0) {
            final Procedure proc = subprocStack.get(stackTail);
      
            LockState lockState;
            //If reuseLock, then don't acquire the lock
            if (!reuseLock && (lockState = acquireLock(proc)) != LockState.LOCK_ACQUIRED) {
              return lockState;
            }
      
            lockState = executeRollback(proc);
            boolean abortRollback = lockState != LockState.LOCK_ACQUIRED;
            abortRollback |= !isRunning() || !store.isRunning();
      
            //If the next procedure in the stack is the current one, then reuseLock = true
            reuseLock = stackTail > 0 && (subprocStack.get(stackTail - 1) == proc) && !abortRollback;
            //If reuseLock, don't releaseLock
            if (!reuseLock) {
              releaseLock(proc, false);
            }
      
            if (abortRollback) {
              return lockState;
            }
      
            subprocStack.remove(stackTail);
      
            if (proc.isYieldAfterExecutionStep(getEnvironment())) {
              return LockState.LOCK_YIELD_WAIT;
            }
      
            //But, here, lock is released no matter reuseLock is true or false
            if (proc != rootProc) {
              execCompletionCleanup(proc);
            }
          }
      

      You can see my comments in the code above, reuseLock can cause the procedure executing(rollback) without a lock. Though I haven't found any bugs introduced by this issue, it is indeed a potential bug need to fix.
      I think we can just remove the reuseLock logic. Acquire and release lock every time.

      Find another case that during rolling back, the procedure's lock may not be released properly:
      see comment: https://issues.apache.org/jira/browse/HBASE-20975?focusedCommentId=16565123&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16565123

      Attachments

        1. HBASE-20975.branch-2.0.001.patch
          2 kB
          Allan Yang
        2. HBASE-20975.branch-2.0.002.patch
          2 kB
          Allan Yang
        3. HBASE-20975.branch-2.0.003.patch
          2 kB
          Allan Yang
        4. HBASE-20975.branch-2.0.004.patch
          2 kB
          Allan Yang
        5. HBASE-20975.branch-2.0.005.patch
          3 kB
          Allan Yang
        6. HBASE-20975.branch-2.1.001.patch
          3 kB
          Allan Yang

        Activity

          People

            allan163 Allan Yang
            allan163 Allan Yang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: