Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-1959

Introduce Gobblin partial success semantic which fails job on partial success

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • gobblin-core
    • None

    Description

      Currently, the partial success semantics in Gobblin where it follows a commit on task success policy will return a successful job running status even though some tasks fail. This is because in most Gobblin jobs, there would be movement of the watermark after the tasks succeed so we want to follow that semantic to not lead to duplicate ingestion.

      However, there are some Gobblin jobs (distcp) which do not follow the watermark semantic, thus a partial success should return a failed status so that the job callers can understand that not every task has succeeded, thus invoke a retry with amended amounts of work.

      Attachments

        Activity

          People

            abti Abhishek Tiwari
            wlo William Lo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10m
                10m