Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24015

Disable query-based compaction on MR execution engine

    XMLWordPrintableJSON

Details

    Description

      Major compaction can be run when the execution engine is MR. This can cause data loss a la HIVE-23703 (the fix for data loss when the execution engine is MR was reverted by HIVE-23763).
      Currently minor compaction can only be run when the execution engine is Tez, otherwise it falls back to MR (non-query-based) compaction. We should extend this functionality to major compaction as well.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-23703 Major QB compaction with multiple FileSinkOperators results in data loss and one original file

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-23763 Query based minor compaction produces wrong files when rows with different buckets Ids are processed by the same FileSinkOperator

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #1375

          Activity

            People

              klcopp Karen Coppage
              klcopp Karen Coppage
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m