Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14841 Replication - Phase 2
  3. HIVE-19499

Bootstrap REPL LOAD shall add tasks to create checkpoints for db/tables/partitions.

    XMLWordPrintableJSON

Details

    Description

      Currently. bootstrap REPL LOAD expect the target database to be empty or not exist to start bootstrap load.

      But, this adds overhead when there is a failure in between bootstrap load and there is no way to resume it from where it fails. So, it is needed to create checkpoints in table/partitions to skip the completely loaded objects.

      Use the fully qualified path of the dump directory as a checkpoint identifier. This should be added to the table / partition properties in hive via a task, as the last task in the DAG for table / partition creation.

      Attachments

        1. HIVE-19499.01.patch
          26 kB
          Sankar Hariappan
        2. HIVE-19499.02.patch
          26 kB
          Sankar Hariappan
        3. HIVE-19499.01-branch-3.patch
          26 kB
          Sankar Hariappan
        4. HIVE-19499.02-branch-3.patch
          26 kB
          Sankar Hariappan

        Issue Links

          Activity

            People

              sankarh Sankar Hariappan
              sankarh Sankar Hariappan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: