Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18320 Support ACID Tables Replication
  3. HIVE-18747

Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.

    XMLWordPrintableJSON

Details

    Description

      Per table write ID implementation (HIVE-18192) maintains a map between txn ID and table write ID in TXN_TO_WRITE_ID meta table. 

      The entries in this table is used to generate ValidWriteIdList for the given ValidTxnList to ensure snapshot isolation. 

      When table or database is dropped, then these entries are cleaned-up. But, it is necessary to clean-up for active tables too for better performance.

      TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID.  The state of each Write ID (open, committed, aborted) is determined by the state of the parent transaction.  In order to be able to get a WriteIdList that is accurate wrt ValidTxnList that is locked in at the start of the transaction, we have to retain txnid<->writeid mapping even after the transaction ends. This is because a reader at Snapshot Isolation that started when transaction X was open, should continue to ignore the data written by X even after X commits.

      So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID.  There are 2 parts to it. When txn X is opened, it records Y=select min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) table, i.e. it adds (X, Y) to MIN_HISTORY.  On commit (and abort) of X, it removes its own entry from MIN_HISTORY. In the absence of Aborted transactions, MIN_HISTORY gives us the smallest open txnid across all active reader snapshots.  Let Z=select min(opentxnid) from MIN_HISTORY. We can delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since every active reader sees txns < Z as committed.

      If S is aborted txns, we retain the metadata about it in TXNS as long as any data written S may be visible to some reader in the system so that the reader knows to skip this data.  The rules for when that is are complex but wrt to TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A).  

      If no open or aborted txns exist in the system, then we need to enable cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next).  

      Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator immediately after cleaning up aborted txns metadata from TXNS table.

       

      Attachments

        1. HIVE-18747.01.patch
          1.39 MB
          Sankar Hariappan
        2. HIVE-18747.02.patch
          38 kB
          Sankar Hariappan
        3. HIVE-18747.03.patch
          39 kB
          Sankar Hariappan
        4. HIVE-18747.04.patch
          39 kB
          Sankar Hariappan
        5. HIVE-18747.05.patch
          39 kB
          Sankar Hariappan
        6. HIVE-18747.06.patch
          40 kB
          Sankar Hariappan

        Issue Links

          Activity

            People

              sankarh Sankar Hariappan
              sankarh Sankar Hariappan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: