Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.3, 3.2.4, 4.0.0, 3.5.1, 3.3.4
-
None
-
None
Description
The current blockmanager decommisssioner uses the periodic threads to refresh blocks/ peers, monitor progress. It can be low-effeicent this way. For example, in the worst case, it takes 30s (by default) at most for an executor to exit itself even if all the blocks have been migrated. The cause is that the migration status is checked every 30s (by default).
So this ticket proposes the blockmanager decommissioner to leverage the events driven way to improve its effeciency.
Attachments
Issue Links
- relates to
-
SPARK-20624 SPIP: Add better handling for node shutdown
- In Progress
1.
|
On-demand shuffle migration peer refresh during decommission | Open | Unassigned |