Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.0
-
None
Description
Similar to MapStatus, which tracks the metadata about each map task's shuffle output, we also need to track the metadata about each merged shuffle partition with push-based shuffle. We currently term this as MergeStatus.
Since MergeStatus tracks metadata from the perspective of reducer tasks, it's not efficient to break up the metadata tracked in a MergeStatus and spread it across multiple MapStatus.