Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-1231

Make re-compaction be able to write to a new folder based on the executCount

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.15.0
    • None
    • None

    Description

      In iceberg world, we should not delete file in a dataset directly, so re-compaction should not delete the old daily data, instead it should write to a new folder. This ticket is to add a config to enable re-compaction write to a new folder instead of overwrite the existing folder. After the configure is enabled, the daily data directory will be /daily/yyyy/mm/dd/compaction_{executionCount}

      Attachments

        Activity

          People

            Unassigned Unassigned
            Zihan Li Zihan Li
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h 10m
                2h 10m