Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Observation: Seems that Tests are currently using HadoopShimsPre2_3 (cause of min.hadoop.version) that does not support a zero copy reader – for testing purposes we probably want to test the latest Shims (including the most up-to-date features) – opening a jira for this.
Bumping min.hadoop.version to 2.7.3 and enabling zero copy reveals an underlying issue in buffer management for the StripePlanner:
For example running: testReadFormat_0_11 with conf.setBoolean(OrcConf.USE_ZEROCOPY.getHiveConfName(), true)
java.lang.IllegalArgumentException: tried to release a buffer that was not created by this stream.
at org.apache.hadoop.fs.FSDataInputStream.releaseBuffer(FSDataInputStream.java:217)
at org.apache.orc.impl.ZeroCopyShims$ZeroCopyAdapter.releaseBuffer(ZeroCopyShims.java:75)
at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.releaseBuffer(RecordReaderUtils.java:134)
at org.apache.orc.impl.reader.StripePlanner$StreamInformation.releaseBuffers(StripePlanner.java:518)
at org.apache.orc.impl.reader.StripePlanner.clearStreams(StripePlanner.java:183)
at org.apache.orc.impl.RecordReaderImpl.clearStreams(RecordReaderImpl.java:1089)
at org.apache.orc.impl.RecordReaderImpl.beginReadStripe(RecordReaderImpl.java:1124)
at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1098)
at org.apache.orc.impl.RecordReaderImpl.seekToRow(RecordReaderImpl.java:1348)
at org.apache.orc.TestVectorOrcFile.testReadFormat_0_11(TestVectorOrcFile.java:341)
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-