Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Looks like we are very conservative wrt memory configs used for spillable map based FSV.
For eg, we are only allocating 15Mb out of the box to file groups when using spillable map based FSV.
public long getMaxMemoryForFileGroupMap() {
long totalMemory = getLong(SPILLABLE_MEMORY);
return totalMemory - getMaxMemoryForPendingCompaction() - getMaxMemoryForBootstrapBaseFile();
}
SPILLABLE_MEMORY = default is 100Mb.
getMaxMemoryForPendingCompaction = 80% of 100MB.
getMaxMemoryForBootstrapBaseFile = 5% of 100Mb.
so, overall, out of the box we are allocating only 15Mb for getMaxMemoryForFileGroupMap.
ref: https://github.com/apache/hudi/blob/bb0621edee97507cf2460e8cb57b5307510b917e/hudi-[…]/apache/hudi/common/table/view/FileSystemViewStorageConfig.java
Wondering do we even need 80% for pending compaction tracker in our FSV. I am thinking to make it 15%. so that we can give more memory to actual file groups. We may not have lot of pending compactions for a given table.
Attachments
Issue Links
- links to