Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3702

Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client

    XMLWordPrintableJSON

Details

    • Hide
      This patch will attempt to allocate all replicas to remote DataNodes, by adding local DataNode to the excluded DataNodes. If no sufficient replicas can be obtained, it will fall back to default block placement policy, which writes one replica to local DataNode.
      Show
      This patch will attempt to allocate all replicas to remote DataNodes, by adding local DataNode to the excluded DataNodes. If no sufficient replicas can be obtained, it will fall back to default block placement policy, which writes one replica to local DataNode.

    Description

      This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are not read when there are no failures.

      Taking HBase as an example, these files will be read only if the process that wrote them (the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only 2 of them are really useful.

      Attachments

        1. HDFS-3702_Design.pdf
          63 kB
          Lei (Eddy) Xu
        2. HDFS-3702.000.patch
          20 kB
          Lei (Eddy) Xu
        3. HDFS-3702.001.patch
          19 kB
          Lei (Eddy) Xu
        4. HDFS-3702.002.patch
          25 kB
          Lei (Eddy) Xu
        5. HDFS-3702.003.patch
          23 kB
          Lei (Eddy) Xu
        6. HDFS-3702.004.patch
          75 kB
          Lei (Eddy) Xu
        7. HDFS-3702.005.patch
          75 kB
          Lei (Eddy) Xu
        8. HDFS-3702.006.patch
          75 kB
          Lei (Eddy) Xu
        9. HDFS-3702.007.patch
          74 kB
          Lei (Eddy) Xu
        10. HDFS-3702.008.patch
          75 kB
          Lei (Eddy) Xu
        11. HDFS-3702.009.patch
          75 kB
          Lei (Eddy) Xu
        12. HDFS-3702.010.patch
          78 kB
          Lei (Eddy) Xu
        13. HDFS-3702.011.patch
          79 kB
          Lei (Eddy) Xu
        14. HDFS-3702.012.patch
          79 kB
          Lei (Eddy) Xu

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-4946 Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-13573 Javadoc for BlockPlacementPolicyDefault is inaccurate

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-4946 Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable

          • Major - Major loss of function.
          • Closed
          relates to

          Umbrella - An overarching type made of sub-tasks HBASE-5843 Improve HBase MTTR - Mean Time To Recover

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #176

          Activity

            People

              eddyxu Lei (Eddy) Xu
              nkeywal Nicolas Liochon
              Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: