Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17524

OIV: add Transformed processor which reconstructs an fsimage from another fsimage file

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0, 3.3.4
    • None
    • tools

    Description

      Background:

      The Image file generated by the existing Hadoop 3.3.4 version cannot be forward compatible . In the high version of HDFS, the fsimage file conversion tool is provided to support the generation of forward compatible fsimage file to support the downgrade operation.

      Description

      Because there are differences in the structure and loading methods of some Sections between high and low versions of fsimage files, especially the StringTable Section. This will make it impossible to downgrade to a lower version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because when the lower version of HDFS loads the fsimage file generated by the higher version of HDFS, there will be an ArrayIndexOutOfBoundsException.

       

      The code differences are as follows:

      // 3.3.4  
      static SerialNumberManager.StringTable loadStringTable(InputStream in)
              throws IOException {
        ··· ···    
        SerialNumberManager.StringTable stringTable =            SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());    
        for (int i = 0; i < s.getNumEntry(); ++i) {
          FsImageProto.StringTableSection.Entry e = FsImageProto          .StringTableSection.Entry.parseDelimitedFrom(in);
          stringTable.put(e.getId(), e.getStr());
        }    
      return stringTable;
      } 
      
      
      // 3.1.1
      static String[] loadStringTable(InputStream in) throws IOException {
        ··· ···
        String[] stringTable = new String[s.getNumEntry() + 1];
        for (int i = 0; i < s.getNumEntry(); ++i) {
          FsImageProto.StringTableSection.Entry e = FsImageProto
              .StringTableSection.Entry.parseDelimitedFrom(in);
          // ArrayIndexOutOfBoundsException is triggered when loading a higher version of the fsimage file.
          stringTable[e.getId()] = e.getStr();
        }
        return stringTable;
      }

      Solution
      Solution Reference from HDFS-17463

      From the figure, it can be seen that the Id arrangement of StringTable in the fsimage file has changed from a compact arrangement to a decentralized arrangement, that is, USER, GROUP and XATTR are no longer mixed. The arrangement is divided into different storage areas and arranged separately.

      • With the sub-sections feature introduced in HDFS-14617, Protobuf can support compatible reading. 
      • When saving fsimage files in high and low versions, the main difference is the arrangement of Entry(e.g., USER, GROUP, and XATTR ) in StringTable.
      • We will add a conversion tool to convert the Id arrangement of the high version fsimage file StringTable to a compact arrangement, so that the low version can be compatible with this format fsimage file.

       

       

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              wuxiaobao Xiaobao Wu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: