Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-614

Implement efficient seek() in decompression streams

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.7.0
    • 1.7.0
    • C++
    • None

    Description

      The current implementation of ZlibDecompressionStream/BlockDecompressionStream::seek resets the state of the decompressor and the underlying file reader and throws away their buffers. The buffers can still have usable data in the following cases;
      1. If the new row group's start position is in the same compressed chunk we were reading, then we just jumped to another position within the same uncompressed buffer, so both the original compressed buffer and the decompressed buffer can be reused. This is a very common scenario with the default ORC configs of unaligned 256KB>=chunks and 10K row groups, e.g. chunk can contain 3 full row groups of 8 byte int without any encoding.
      2. If the new row group's start position is in another compressed chunk, but it starts in the current compressed buffer (as we have read ahead during file reading), then the compressed buffer can be kept and only the uncompressed buffer needs to be dropped. This is the usual case in Apache Impala, as 8 MB block size is used which leads to reading the whole stream to the buffer for typical columns.

      The lack of these optimizations lead to regression during the testing of https://github.com/apache/orc/pull/476, which uses seek() when a row group is skipped due to predicate push down, as all seeks caused the whole stream to be read again.

      Attachments

        Issue Links

          Activity

            People

              wgtmac Gang Wu
              csringhofer Csaba Ringhofer
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m