Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Currently, the partial success semantics in Gobblin where it follows a commit on task success policy will return a successful job running status even though some tasks fail. This is because in most Gobblin jobs, there would be movement of the watermark after the tasks succeed so we want to follow that semantic to not lead to duplicate ingestion.
However, there are some Gobblin jobs (distcp) which do not follow the watermark semantic, thus a partial success should return a failed status so that the job callers can understand that not every task has succeeded, thus invoke a retry with amended amounts of work.