Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.2.0, 3.3.4
-
None
Description
Background:
The Image file generated by the existing Hadoop 3.3.4 version cannot be forward compatible . In the high version of HDFS, the fsimage file conversion tool is provided to support the generation of forward compatible fsimage file to support the downgrade operation.
Description:
Because there are differences in the structure and loading methods of some Sections between high and low versions of fsimage files, especially the StringTable Section. This will make it impossible to downgrade to a lower version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because when the lower version of HDFS loads the fsimage file generated by the higher version of HDFS, there will be an ArrayIndexOutOfBoundsException.
The code differences are as follows:
// 3.3.4 static SerialNumberManager.StringTable loadStringTable(InputStream in) throws IOException { ··· ··· SerialNumberManager.StringTable stringTable = SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits()); for (int i = 0; i < s.getNumEntry(); ++i) { FsImageProto.StringTableSection.Entry e = FsImageProto .StringTableSection.Entry.parseDelimitedFrom(in); stringTable.put(e.getId(), e.getStr()); } return stringTable; } // 3.1.1 static String[] loadStringTable(InputStream in) throws IOException { ··· ··· String[] stringTable = new String[s.getNumEntry() + 1]; for (int i = 0; i < s.getNumEntry(); ++i) { FsImageProto.StringTableSection.Entry e = FsImageProto .StringTableSection.Entry.parseDelimitedFrom(in); // ArrayIndexOutOfBoundsException is triggered when loading a higher version of the fsimage file. stringTable[e.getId()] = e.getStr(); } return stringTable; }
Solution:
Solution Reference from HDFS-17463
From the figure, it can be seen that the Id arrangement of StringTable in the fsimage file has changed from a compact arrangement to a decentralized arrangement, that is, USER, GROUP and XATTR are no longer mixed. The arrangement is divided into different storage areas and arranged separately.
- With the sub-sections feature introduced in
HDFS-14617, Protobuf can support compatible reading. - When saving fsimage files in high and low versions, the main difference is the arrangement of Entry(e.g., USER, GROUP, and XATTR ) in StringTable.
- We will add a conversion tool to convert the Id arrangement of the high version fsimage file StringTable to a compact arrangement, so that the low version can be compatible with this format fsimage file.
Attachments
Issue Links
- links to