Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6521

Bypass network shuffle read if both endpoints are local

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 1.2.0
    • None
    • Shuffle, Spark Core
    • None

    Description

      In the past, executor read other executor's shuffle file in the same node by net. This pr make that executors in the same node read local shuffle file In sort-based Shuffle. It will reduce net transport.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-27651 Avoid the network when block manager fetches shuffle blocks from the same host

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #5178 [Github] Pull Request #5178 (viper-kun)

          Pull request #9478 [Github] Pull Request #9478 (maropu)

          Activity

            People

              Unassigned Unassigned
              xukun xukun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: