Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27677

Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.3
    • 3.0.0
    • Block Manager, Spark Core
    • None

    Description

      Disk-cached RDD blocks are currently unavailable after an executor is removed. However, when there is an external shuffle service, the data remains available on disk and could be served by the shuffle service. This would allow dynamic allocation to reclaim executors with only disk-cached blocks more rapidly, but still keep the cached data available.

      Attachments

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. SPARK-36446 YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-25888 Service requests for persist() blocks via external service after dynamic deallocation

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #24499 [Github] Pull Request #24499 (attilapiros)

          Pull request #24499 [Github] Pull Request #24499 (attilapiros)

          Web Link GitHub Pull Request #24499

          Web Link GitHub Pull Request #24697

          Activity

            People

              attilapiros Attila Zsolt Piros
              irashid Imran Rashid
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: