Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.0
-
None
Description
In SPARK-23243 and SPARK-25341, the concept of an INDETERMINATE stage was introduced, which would be handled differently if retried.
Since these was added to address a data correctness issue, we should also add support for these in push-based shuffle, so that we would be able to rollback the merged shuffle partitions of a shuffle map stage if it's an INDETERMINATE stage.