Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6584

Support Archival Storage

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • balancer & mover, namenode
    • None
    • Reviewed

    Description

      In most of the Hadoop clusters, as more and more data is stored for longer time, the demand for storage is outstripping the compute. Hadoop needs a cost effective and easy to manage solution to meet this demand for storage. Current solution is:

      • Delete the old unused data. This comes at operational cost of identifying unnecessary data and deleting them manually.
      • Add more nodes to the clusters. This adds along with storage capacity unnecessary compute capacity to the cluster.

      Hadoop needs a solution to decouple growing storage capacity from compute capacity. Nodes with higher density and less expensive storage with low compute power are becoming available and can be used as cold storage in the clusters. Based on policy the data from hot storage can be moved to cold storage. Adding more nodes to the cold storage can grow the storage independent of the compute capacity in the cluster.

      Attachments

        1. archival-storage-testplan.pdf
          45 kB
          Jing Zhao
        2. h6584_20140907.patch
          372 kB
          Tsz-wo Sze
        3. h6584_20140908.patch
          373 kB
          Tsz-wo Sze
        4. h6584_20140908b.patch
          388 kB
          Jing Zhao
        5. h6584_20140911.patch
          395 kB
          Tsz-wo Sze
        6. h6584_20140911b.patch
          394 kB
          Jing Zhao
        7. h6584_20140915.patch
          397 kB
          Jing Zhao
        8. h6584_20140916.patch
          400 kB
          Jing Zhao
        9. h6584_20140916.patch
          402 kB
          Jing Zhao
        10. h6584_20140917.patch
          420 kB
          Jing Zhao
        11. h6584_20140917b.patch
          420 kB
          Jing Zhao
        12. h6584_20140918.patch
          425 kB
          Tsz-wo Sze
        13. h6584_20140918b.patch
          423 kB
          Tsz-wo Sze
        14. HDFS-6584.000.patch
          344 kB
          Jing Zhao
        15. HDFSArchivalStorageDesign20140623.pdf
          152 kB
          Tsz-wo Sze
        16. HDFSArchivalStorageDesign20140715.pdf
          166 kB
          Tsz-wo Sze

        Issue Links

          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-7197 Enhancements to Mover

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-13678 StorageType is incompatible when rolling upgrade to 2.6/2.6+ versions

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7147 Update archival storage user documentation

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-7076 Allow users to define custom storage policies

          • Major - Major loss of function.
          • Patch Available
          relates to

          Sub-task - The sub-task of the issue HDFS-7184 Allow data migration tool to run as a daemon

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-7285 Erasure Coding Support inside HDFS

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-5682 Heterogeneous Storage phase 2 - APIs to expose Storage Types

          • Major - Major loss of function.
          • Closed

          Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-6572 Tiered HFile storage

          • Major - Major loss of function.
          • Closed
          requires

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6702 DFSClient should create blocks using StorageType

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6685 Balancer should preserve storage type of replicas

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6700 BlockPlacementPolicy shoud choose storage but not datanode for deletion

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          There are no Sub-Tasks for this issue.

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              39 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: