Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7955

Dynamic allocation: longer timeout for executors with cached blocks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.4.0
    • 1.4.1, 1.5.0
    • Spark Core
    • None

    Description

      When dynamic allocation is enabled, executor idle time is currently the only parameter considered. This can be annoying if executors get removed but have cached blocks. This can cause sever performance degradation.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-9197 Cached RDD partitions are lost when executors are dynamically deallocated

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-4280 In dynamic allocation, add option to never kill executors with cached blocks

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20624 SPIP: Add better handling for node shutdown

          • Major - Major loss of function.
          • In Progress
          links to

          Pull request #6508 [Github] Pull Request #6508 (harishreedharan)

          Activity

            People

              hshreedharan Hari Shreedharan
              hshreedharan Hari Shreedharan
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: