Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1804

Add a new block-volume device choosing policy that looks at free space

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.0.4-alpha
    • 2.1.0-beta
    • datanode
    • None
    • Reviewed
    • Hide
      There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".
      Show
      There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".

    Description

      HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.

      An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.

      Attachments

        1. HDFS-1804.patch
          31 kB
          Aaron Myers
        2. HDFS-1804.patch
          31 kB
          Aaron Myers
        3. HDFS-1804.patch
          31 kB
          Aaron Myers
        4. HDFS-1804.patch
          29 kB
          Aaron Myers
        5. HDFS-1804.patch
          28 kB
          Aaron Myers

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-1312 Re-balance disks within a Datanode

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1120 Make DataNode's block-to-device placement policy pluggable

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              atm Aaron Myers
              qwertymaniac Harsh J
              Votes:
              0 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: