Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.2.0, 3.3.5, 3.3.3, 3.3.4
-
None
Description
desc:
Hadoop 3.2 introduced optimization features for HDFS StringTable (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of Hadoop upgraded to 3.2 and later versions not supporting downgrade operations.
- This issue has also been discussed in
HDFS-14831, and it is recommended to revert the feature, but it cannot fundamentally solve the problem。
Therefore, we have added an optimization to support downgrading
Solution:
- First, we will add the "dfs. image. save. splitId. stringTable" conf switch "StringTable optimization feature" is enabled
- When the conf value is false, an Image file compatible with lower versions of HDFS is generated to support downgrading.
The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 is shown in the following figure.- With the sub-sections feature introduced in
HDFS-14617, Protobuf can support compatible reading.
The data structure causing incompatible differences is mainly StringTable.
- In "dfs.image.save.splitId.stringTable = false " the Id generation order of StringTable starts from 0 to Integer.Max. When true, the Id value range follows the latest rules.
Attachments
Attachments
Issue Links
- links to