Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20738 Enable Delete Event filtering in VectorizedOrcAcidRowBatchReader
  3. HIVE-16812

VectorizedOrcAcidRowBatchReader doesn't filter delete events

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.3.0
    • 4.0.0-alpha-1
    • Transactions
    • None
    • n/a

    Description

      the c'tor of VectorizedOrcAcidRowBatchReader has

          // Clone readerOptions for deleteEvents.
    Reader.Options deleteEventReaderOptions = readerOptions.clone();
    // Set the range on the deleteEventReaderOptions to 0 to INTEGER_MAX because
    // we always want to read all the delete delta files.
    deleteEventReaderOptions.range(0, Long.MAX_VALUE);

      This is suboptimal since base and deltas are sorted by ROW__ID. So for each split if base we can find min/max ROW_ID and only load events from delta that are in [min,max] range. This will reduce the number of delete events we load in memory (to no more than there in the split).

      When we support sorting on PK, the same should apply but we'd need to make sure to store PKs in ORC index

      See OrcRawRecordMerger.discoverKeyBounds()

      hive.acid.key.index in Orc footer has an index of ROW__IDs so we should know min/max easily for any file written by OrcRecordUpdater

      Attachments

        1. HIVE-16812.07.patch
          74 kB
          Eugene Koifman
        2. HIVE-16812.06.patch
          74 kB
          Eugene Koifman
        3. HIVE-16812.05.patch
          70 kB
          Eugene Koifman
        4. HIVE-16812.04.patch
          61 kB
          Eugene Koifman
        5. HIVE-16812.02.patch
          30 kB
          Eugene Koifman

        Issue Links

          blocks

          Sub-task - The sub-task of the issue HIVE-20635 VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. HIVE-22318 Java.io.exception:Two readers for

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HIVE-23143 Transactions: PPD in Delete deltas is broken

          • Major - Major loss of function.
          • Open
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-18662 hive.acid.key.index is missing entries

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue HIVE-20694 Additional unit tests for VectorizedOrcAcidRowBatchReader min max key evaluation

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-17320 OrcRawRecordMerger.discoverKeyBounds logic can be simplified

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-17284 remove OrcRecordUpdater.deleteEventIndexBuilder

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-17458 VectorizedOrcAcidRowBatchReader doesn't handle 'original' files

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HIVE-17231 ColumnizedDeleteEventRegistry.DeleteReaderValue optimization

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HIVE-14233 Improve vectorization for ACID by eliminating row-by-row stitching

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-19985 ACID: Skip decoding the ROW__ID sections for read-only queries

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-20604 Minor compaction disables ORC column stats

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Review Board

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: