Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-4656

Add a container balancer tool or service for HDDS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.2.0
    • Ozone Datanode, SCM, Tools
    • None

    Description

      When an existing Ozone cluster is nearly full, we have to add more datanodes into the Ozone cluster, but there are two issue we must face.

      • When new allocate container request coming, SCM should better to choose the datanodes in low usage, if not, the performance will getting pool.
      • For read request, the existing datanodes stored lots of blocks, so they are responsible for serving the read request and supply the data stream service, meanwhile, the new coming datanodes can help nothing.

      If we have a balancer tool just like hdfs balancer, we can move the block or container from some high usage datanodes to low, I think this is one of necessary tools for Ozone.

      container balancer design doc https://docs.google.com/document/d/15PdYaP6aLB18ptbcOK3XWlL4Y1PLfIP-ll30KjNPZ2g/edit?usp=sharing

      Attachments

        1. Container Balancer Design.pdf
          128 kB
          Lokesh Jain
        2. Container Move.pdf
          62 kB
          Lokesh Jain
        3. Ozone Balancer HA.pdf
          651 kB
          Siddhant Sangwan
        4. Replication Manager V2.pdf
          149 kB
          Siddhant Sangwan
        5. Ratis vs EC for Container Balancer.pdf
          50 kB
          Siddhant Sangwan
        There are no Sub-Tasks for this issue.

        Activity

          People

            maobaolong Baolong Mao
            maobaolong Baolong Mao
            Votes:
            3 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: