Details
-
New Feature
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
3.2.0
-
None
-
None
Description
Shuffle data corruption is a long-standing issue in Spark. For example, inĀ SPARK-18105, people continually reports corruption issue. However, data corruption is difficult to reproduce in most cases and even harder to tell the root cause. We don't know if it's a Spark issue or not. With the checksum support for the shuffle, Spark itself can at least distinguish the cause between disk and network, which is very important for users.
Attachments
Issue Links
- is related to
-
SPARK-49459 Support CRC32C for Shuffle Checksum
- Resolved
- links to
1.
|
Add shuffle checksum support for push-based shuffle | Open | Unassigned |