Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10710

In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved with NN lock protection

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • namenode
    • None
    • Reviewed

    Description

      In BlockManager#rescanPostponedMisreplicatedBlocks(), start and end block counts should be get with the protect with lock. Or, log records like "-1 blocks are removed" which indicate minus blocks are removed could be generated.

      For example, following scenario:
      1. thread1 run long startPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount(); currently startPostponedMisReplicatedBlocksCount get the value 20.
      2. before thread1 run namesystem.writeLock(); , thread2 increment postponedMisreplicatedBlocksCount by 1, so postponedMisreplicatedBlocksCount is 21 now.
      3. thread1 end the iteration, but no postponed block is removed, so after run long endPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount();, endPostponedMisReplicatedBlocksCount get the value of 21.
      4. thread 1 generate the log:

            LOG.info("Rescan of postponedMisreplicatedBlocks completed in " +
                (Time.monotonicNow() - startTimeRescanPostponedMisReplicatedBlocks) +
                " msecs. " + endPostponedMisReplicatedBlocksCount +
                " blocks are left. " + (startPostponedMisReplicatedBlocksCount -
                endPostponedMisReplicatedBlocksCount) + " blocks are removed.");
      

      Then, we'll get the log record like "-1 blocks are removed."

      Attachments

        1. HDFS-10710.1.patch
          1 kB
          Rui Gao

        Activity

          People

            demongaorui Rui Gao
            demongaorui Rui Gao
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: