Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-48636

Event driven block manager decommissioner

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.3, 3.2.4, 4.0.0, 3.5.1, 3.3.4
    • None
    • Spark Core
    • None

    Description

      The current blockmanager decommisssioner uses the periodic threads to refresh blocks/ peers, monitor progress. It can be low-effeicent this way. For example, in the worst case, it takes 30s (by default) at most for an executor to exit itself even if all the blocks have been migrated. The cause is that the migration status is checked every 30s (by default).

      So this ticket proposes the blockmanager decommissioner to leverage the events driven way to improve its effeciency. 

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Ngone51 wuyi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: