Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15493

Update block map and name cache in parallel while loading fsimage.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.1, 3.4.0
    • 3.3.1, 3.4.0
    • namenode
    • None
    • Reviewed

    Description

      While loading INodeDirectorySection of fsimage, it will update name cache and block map after added inode file to inode directory. It would reduce time cost of fsimage loading to enable these steps run in parallel.

      In our test case, with patch HDFS-13694 and HDFS-14617, the time cost to load fsimage (220M files & 240M blocks) is 470s, with this patch , the time cost reduce to 410s.

      Attachments

        1. HDFS-15493.001.patch
          9 kB
          Chengwei Wang
        2. fsimage-loading.log
          4 kB
          Chengwei Wang
        3. HDFS-15493.002.patch
          10 kB
          Chengwei Wang
        4. HDFS-15493.003.patch
          10 kB
          Chengwei Wang
        5. HDFS-15493.004.patch
          12 kB
          Chengwei Wang
        6. HDFS-15493.005.patch
          10 kB
          Chengwei Wang
        7. HDFS-15493.006.patch
          11 kB
          Chengwei Wang
        8. HDFS-15493.007.patch
          12 kB
          Chengwei Wang
        9. HDFS-15493.008.patch
          12 kB
          Stephen O'Donnell

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-13694 Making md5 computing being in parallel with image loading

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14617 Improve fsimage load time by writing sub-sections to the fsimage index

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              smarthan Chengwei Wang
              smarthan Chengwei Wang
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: