Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30602 SPIP: Support push-based shuffle to improve shuffle efficiency
  3. SPARK-32922

Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.2.0
    • Shuffle, Spark Core
    • None

    Description

      With the extended MapOutputTracker, the reducers can now get the task input data from the merged shuffle partitions for more efficient shuffle data fetch. The reducers should also be able to fallback to fetching the original unmarked blocks if it encounters failures when fetching the merged shuffle partitions.

      Attachments

        Issue Links

          Dependent

          Sub-task - The sub-task of the issue SPARK-35671 Add Support in the ESS to serve merged shuffle block meta and data to executors

          • Major - Major loss of function.
          • Resolved
          requires

          Sub-task - The sub-task of the issue SPARK-33350 Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #30768 [Github] Pull Request #30768 (otterc)

          Pull request #30768 [Github] Pull Request #30768 (otterc)

          Pull request #32140 [Github] Pull Request #32140 (otterc)

          Pull request #32140 [Github] Pull Request #32140 (otterc)

          Pull request #33378 [Github] Pull Request #33378 (otterc)

          Pull request #33378 [Github] Pull Request #33378 (otterc)

          Activity

            People

              csingh Chandni Singh
              mshen Min Shen
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: