Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8999

Allow a file to be closed with COMMITTED but not yet COMPLETE blocks.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • namenode
    • None
    • Reviewed

    Description

      This comes out of a discussion in HDFS-8763. Pasting jingzhao's comment from the jira:

      ...whether we need to let NameNode wait for all the block_received msgs to announce the replica is safe. Looking into the code, now we have

      1. NameNode knows the DataNodes involved when initially setting up the writing pipeline
      2. If any DataNode fails during the writing, client bumps the GS and finally reports all the DataNodes included in the new pipeline to NameNode through the updatePipeline RPC.
      3. When the client received the ack for the last packet of the block (and before the client tries to close the file on NameNode), the replica has been finalized in all the DataNodes.

      Then in this case, when NameNode receives the close request from the client, the NameNode already knows the latest replicas for the block. Currently the checkReplication call only counts in all the replicas that NN has already received the block_received msg, but based on the above #2 and #3, it may be safe to also count in all the replicas in the BlockUnderConstructionFeature#replicas?

      Attachments

        1. h8999_20151228.patch
          18 kB
          Tsz-wo Sze
        2. h8999_20160106.patch
          25 kB
          Tsz-wo Sze
        3. h8999_20160106b.patch
          27 kB
          Tsz-wo Sze
        4. h8999_20160106c.patch
          27 kB
          Tsz-wo Sze
        5. h8999_20160111.patch
          36 kB
          Tsz-wo Sze
        6. h8999_20160113.patch
          38 kB
          Tsz-wo Sze
        7. h8999_20160114.patch
          37 kB
          Tsz-wo Sze
        8. h8999_20160121.patch
          37 kB
          Tsz-wo Sze
        9. h8999_20160121b.patch
          36 kB
          Jing Zhao
        10. h8999_20160121c.patch
          37 kB
          Tsz-wo Sze
        11. h8999_20160121c_branch-2.patch
          40 kB
          Tsz-wo Sze
        12. h8999_20160121c_branch-2.patch
          33 kB
          Tsz-wo Sze

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1172 Blocks in newly completed files are considered under-replicated too quickly

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10810 Setreplication removing block from underconstrcution temporarily when batch IBR is enabled.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-8344 NameNode doesn't recover lease for files with missing blocks

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9710 Change DN to send block receipt IBRs in batches

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15359 EC: Allow closing a file with committed blocks

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15359 EC: Allow closing a file with committed blocks

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              szetszwo Tsz-wo Sze
              jnp Jitendra Nath Pandey
              Votes:
              0 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: