Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1652

Rebalance data blocks when new data nodes added or data nodes become full

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.13.0
    • 0.16.0
    • None
    • None

    Description

      When a new data node joins hdfs cluster, it does not hold much data. So any map task assigned to the machine most likely does not read local data, thus increasing the use of network bandwidth. On the other hand, when some data nodes become full, new data blocks are placed on only non-full data nodes, thus reducing their read parallelism.

      This jira aims to find an approach to redistribute data blocks when imbalance occurs in the cluster. An solution should meet the following requirements:
      1. It maintains data availablility guranteens in the sense that rebalancing does not reduce the number of replicas that a block has or the number of racks that the block resides.
      2. An adminstrator should be able to invoke and interrupt rebalancing from a command line.
      3. Rebalancing should be throttled so that rebalancing does not cause a namenode to be too busy to serve any incoming request or saturate the network.

      Attachments

        1. Balancer.html
          20 kB
          Ravi Phulari
        2. balancer8.patch
          72 kB
          Hairong Kuang
        3. balancer7.patch
          72 kB
          Hairong Kuang
        4. balancer6.patch
          71 kB
          Hairong Kuang
        5. BalancerUserGuide2.pdf
          14 kB
          Hairong Kuang
        6. balancer5.patch
          71 kB
          Hairong Kuang
        7. balancer4.patch
          71 kB
          Hairong Kuang
        8. balancer3.patch
          75 kB
          Hairong Kuang
        9. BalancerAdminGuide1.pdf
          14 kB
          Hairong Kuang
        10. balancer2.patch
          71 kB
          Hairong Kuang
        11. balancer1.patch
          65 kB
          Hairong Kuang
        12. BalancerAdminGuide.pdf
          13 kB
          Hairong Kuang
        13. balancer.patch
          51 kB
          Hairong Kuang
        14. RebalanceDesign6.pdf
          50 kB
          Hairong Kuang
        15. RebalanceDesign5.pdf
          45 kB
          Hairong Kuang
        16. RebalanceDesign4.pdf
          47 kB
          Hairong Kuang

        Issue Links

          depends upon

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-1846 DatanodeReport should distinguish live datanodes from dead datanodes

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-1912 Datanode should support block replacement

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-1914 HDFS should have a NamenodeProtocol to allow secondary namenodes and rebalancing processes to communicate with a primary namenode

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-1266 Remove DatanodeDescriptor dependency from NetworkTopology

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-57 [hbase] Master should allocate regions to regionservers based upon data locality and rack awareness

          • Major - Major loss of function.
          • Closed

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: