Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
During close, the native ORC writer would flush itself - emptying its buffers and leading to the memory estimate to be extremely minimal, then record its memory estimate in the saved state if it has not recorded an estimate before. So for datasets that write a small number of records to file it would greatly underestimate the memory within the native ORC writer.