Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30602 SPIP: Support push-based shuffle to improve shuffle efficiency
  3. SPARK-32916

Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • Shuffle, Spark Core, YARN
    • None

    Description

      Integration needed to bootstrap external shuffle service in YARN deployment mode. Properly create the necessary dirs and initialize the relevant server-side components in the RPC layer.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-34840 Fix cases of corruption in merged shuffle blocks that are pushed

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #30062 [Github] Pull Request #30062 (otterc)

          Pull request #30349 [Github] Pull Request #30349 (otterc)

          Pull request #30433 [Github] Pull Request #30433 (otterc)

          Pull request #30433 [Github] Pull Request #30433 (otterc)

          Pull request #31934 [Github] Pull Request #31934 (otterc)

          Activity

            People

              csingh Chandni Singh
              mshen Min Shen
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: