Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.4.0, 3.5.1
-
None
-
None
Description
As part of SPARK-39195, task is marked as failed but the exception chain was not sent, ultimately the cause becomes `null` in SparkException. It is not convenient to find the root cause from the detailed message.
/** * Called by the OutputCommitCoordinator to cancel stage due to data duplication may happen. */ private[scheduler] def stageFailed(stageId: Int, reason: String): Unit = { eventProcessLoop.post(StageFailed(stageId, reason, None)) }
Attachments
Issue Links
- is a clone of
-
SPARK-39868 StageFailed event should attach with the root cause
- Resolved
- Parent Feature
-
SPARK-39195 Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
- Resolved