Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1606

Provide a stronger data guarantee in the write pipeline

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.23.0
    • 0.23.0
    • datanode, hdfs-client, namenode
    • None
    • Incompatible change, Reviewed
    • Hide
      Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy. Added a new feature to replace datanode on failure in DataTransferProtocol. Added getAdditionalDatanode(..) in ClientProtocol.
      Show
      Added two configuration properties, dfs.client.block.write.replace-datanode-on-failure.enable and dfs.client.block.write.replace-datanode-on-failure.policy. Added a new feature to replace datanode on failure in DataTransferProtocol. Added getAdditionalDatanode(..) in ClientProtocol.

    Description

      In the current design, if there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. Unfortunately, it is possible that DFSClient may incorrectly remove a healthy datanode but leave the failed datanode in the pipeline because failure detection may be inaccurate under erroneous conditions.

      We propose to have a new mechanism for adding new datanodes to the pipeline in order to provide a stronger data guarantee.

      Attachments

        1. h1606_20110408b.patch
          52 kB
          Tsz-wo Sze
        2. h1606_20110408.patch
          49 kB
          Tsz-wo Sze
        3. h1606_20110407c.patch
          50 kB
          Tsz-wo Sze
        4. h1606_20110407b.patch
          48 kB
          Tsz-wo Sze
        5. h1606_20110407.patch
          48 kB
          Tsz-wo Sze
        6. h1606_20110406b.patch
          43 kB
          Tsz-wo Sze
        7. h1606_20110406.patch
          41 kB
          Tsz-wo Sze
        8. h1606_20110405b.patch
          35 kB
          Tsz-wo Sze
        9. h1606_20110405.patch
          35 kB
          Tsz-wo Sze
        10. h1606_20110404.patch
          34 kB
          Tsz-wo Sze
        11. h1606_20110228.patch
          34 kB
          Tsz-wo Sze
        12. h1606_20110217.patch
          25 kB
          Tsz-wo Sze
        13. h1606_20110211.patch
          20 kB
          Tsz-wo Sze
        14. h1606_20110210.patch
          7 kB
          Tsz-wo Sze

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1595 DFSClient may incorrectly detect datanode failure

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1599 Umbrella Jira for Improving HBASE support in HDFS

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-265 Revisit append

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1785 Cleanup BlockReceiver and DataXceiver

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1789 Refactor frequently used codes from DFSOutputStream, BlockReceiver and DataXceiver

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1817 Split TestFiDataTransferProtocol.java into two files

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: