Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Thrift 0.9.2 removed setReadLength. This causes parquet-thrift to fail because it is called for TBinaryProtocol. The reason we use it is defensive: a size is read from the data and then that many bytes are read, so using this method sets a maximum and causes an exception rather than a strange failure later on. The code also has a comment that says it is okay when it can't be used.
/* Reduce the chance of OOM when data is corrupted. When readBinary is called on TBinaryProtocol, it reads the length of the binary first,
so if the data is corrupted, it could read a big integer as the length of the binary and therefore causes OOM to happen.
Currently this fix only applies to TBinaryProtocol which has the setReadLength defined.
*/
if (protocol instanceof TBinaryProtocol) {
((TBinaryProtocol)protocol).setReadLength(record.getLength());
}
I think the fix is to remove the section above.