Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1435

Provide an option to store fsimage compressed

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.22.0
    • namenode
    • None
    • Incompatible change, Reviewed
    • This provides an option to store fsimage compressed. The layout version is bumped to -25. The user could configure if s/he wants the fsimage to be compressed or not and which codec to use. By default the fsimage is not compressed.

    Description

      Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network bandwidth when secondary NN uploads a new fsimage to primary NN.

      If we could store fsimage compressed, the problem could be greatly alleviated.

      I plan to provide a new configuration hdfs.image.compressed with a default value of false. If it is set to be true, fsimage is stored as compressed.

      The fsimage will have a new layout with a new field "compressed" in its header, indicating if the namespace is stored compressed or not.

      Attachments

        1. checkpoint-limitandcompress.patch
          13 kB
          Yilei Lu
        2. trunkImageCompress.patch
          14 kB
          Hairong Kuang
        3. trunkImageCompress1.patch
          16 kB
          Hairong Kuang
        4. trunkImageCompress2.patch
          17 kB
          Hairong Kuang
        5. trunkImageCompress3.patch
          17 kB
          Hairong Kuang
        6. trunkImageCompress4.patch
          17 kB
          Hairong Kuang

        Issue Links

          incorporates

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1614 Provide an option to saveNamespace to save namespace uncompressed

          • Major - Major loss of function.
          • Open
          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-6996 Allow CodecFactory to return a codec object given a codec' class name

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1458 Improve checkpoint performance by avoiding unnecessary image downloads

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1457 Limit transmission rate when transfering image between primary and secondary NNs

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1465 Eliminate FS image loading code duplication between OIV and FSImage

          • Major - Major loss of function.
          • Open

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: