Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5535 Umbrella jira for improved HDFS rolling upgrades
  3. HDFS-5754

Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Currently, LayoutVersion defines the on-disk data format and supported features of the entire cluster including NN and DNs. LayoutVersion is persisted in both NN and DNs. When a NN/DN starts up, it checks its supported LayoutVersion against the on-disk LayoutVersion. Also, a DN with a different LayoutVersion than NN cannot register with the NN.

      We propose to split LayoutVersion into two independent values that are local to the nodes:

      • NamenodeLayoutVersion - defines the on-disk data format in NN, including the format of FSImage, editlog and the directory structure.
      • DatanodeLayoutVersion - defines the on-disk data format in DN, including the format of block data file, metadata file, block pool layout, and the directory structure.

      The LayoutVersion check will be removed in DN registration. If NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling upgrade, then only rollback is supported and downgrade is not.

      Attachments

        1. HDFS-5754.013.patch
          141 kB
          Brandon Li
        2. HDFS-5754.012.patch
          139 kB
          Brandon Li
        3. HDFS-5754.010.patch
          138 kB
          Brandon Li
        4. HDFS-5754.009.patch
          138 kB
          Brandon Li
        5. FeatureInfo.patch
          24 kB
          Tsz-wo Sze
        6. HDFS-5754.008.patch
          23 kB
          Brandon Li
        7. HDFS-5754.007.patch
          76 kB
          Brandon Li
        8. HDFS-5754.006.patch
          88 kB
          Brandon Li
        9. HDFS-5754.004.patch
          86 kB
          Brandon Li
        10. HDFS-5754.003.patch
          17 kB
          Brandon Li
        11. HDFS-5754.002.patch
          17 kB
          Brandon Li
        12. HDFS-5754.001.patch
          17 kB
          Brandon Li

        Issue Links

          Activity

            People

              brandonli Brandon Li
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: