[HIVE-20823] Make Compactor run in a transaction - ASF JIRA

XML

Word

Printable

JSON

Have compactor open a transaction and run the job in that transaction.

make compactor produced base/delta include this txn id in the folder name, e.g. base_7_c17 where 17 is the txnid.
add CQ_TXN_ID bigint to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to record this txn id
make sure AcidUtils.getAcidState() pays attention to this transaction on read and ignores this dir if this txn id is not committed in the current snapshot
1. this means not only validWriteIdList but ValidTxnIdList should be passed along in config (if it isn't yet)
once this is done, CompactorMR.createCompactorMarker() can be eliminated and AcidUtils.isValidBase modified accordingly
modify Cleaner so that it doesn't clean old files until new file is visible to all readers

causes

HIVE-21213 Acid table bootstrap replication needs to handle directory created by compaction with txn id

is depended upon by

HIVE-22755 Cleaner/Compaction can skip the read locks and use the min open txn id

is required by

HIVE-20960 Make MM compactor run in a transaction and remove CompactorMR.createCompactorMarker()

relates to

HIVE-20948 Eliminate file rename in compactor

HIVE-20901 running compactor when there is nothing to do produces duplicate data

HIVE-20699 Query based compactor for full CRUD Acid tables

supercedes

HIVE-18772 Make Acid Cleaner use MIN_HISTORY_LEVEL

links to

Review Board

(1 relates to, 1 supercedes, 1 links to)