Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20738 Enable Delete Event filtering in VectorizedOrcAcidRowBatchReader
  3. HIVE-17231

ColumnizedDeleteEventRegistry.DeleteReaderValue optimization

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • Transactions
    • None
    • n/a

    Description

      For unbucketed tables DeleteReaderValue will currently return all delete events. Once we trust that
      the N in bucketN for "base" spit is reliable, all delete events not matching N can be skipped.

      This is useful to protect against extreme cases where someone runs an update/delete on a partition that matches 10 billion rows thus generates very many delete events.

      Since HIVE-19890 all acid data files must have bucketid/writerid in the file name match bucketid/writerid in ROW__ID in the data.

      OrcRawRecrodMerger.getDeltaFiles() should only return files representing the right bucket

      Attachments

        1. HIVE-17231.01.patch
          3 kB
          Eugene Koifman
        2. HIVE-17231.02.patch
          7 kB
          Eugene Koifman

        Issue Links

          contains

          Bug - A problem which impairs or prevents the functions of the product. HIVE-21710 Minor compaction writes delete records in unbucketed tables multiple times when we have multiple files <bucket_N>

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue HIVE-16812 VectorizedOrcAcidRowBatchReader doesn't filter delete events

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          requires

          Bug - A problem which impairs or prevents the functions of the product. HIVE-19890 ACID: Inherit bucket-id from original ROW_ID for delete deltas

          • Major - Major loss of function.
          • Closed
          supercedes

          Sub-task - The sub-task of the issue HIVE-20579 VectorizedOrcAcidRowBatchReader.checkBucketId() should run for unbucketed tables

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: