Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.0
-
None
Description
With push-based shuffle, we are currently decoupling map task executions from the shuffle block push process. Thus, when all map tasks finish, we might want to wait for some small extra time to allow more shuffle blocks to get pushed and merged. This requires some extra coordination in the Spark driver when it transitions from a shuffle map stage to the corresponding reduce stage.
Attachments
Issue Links
- links to
(3 links to)