Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3485

FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      FairScheduler's headroom calculations consider the fairshare and cluster-available-resources, and the fairshare has maxResources. However, for Fifo and Fairshare policies, the fairshare is used only for memory and not cpu. So, the scheduler ends up showing a higher headroom than is actually available. This could lead to applications waiting for resources far longer than then intend to. e.g. MAPREDUCE-6302.

      Attachments

        1. yarn-3485-3.patch
          13 kB
          Karthik Kambatla
        2. yarn-3485-2.patch
          13 kB
          Karthik Kambatla
        3. yarn-3485-1.patch
          13 kB
          Karthik Kambatla
        4. yarn-3485-prelim.patch
          8 kB
          Karthik Kambatla

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. YARN-3891 Apps are starved for resource due to default FairSharePolicy in Fair Scheduler

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6302 Preempt reducers after a configurable timeout irrespective of headroom

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              kasha Karthik Kambatla
              kasha Karthik Kambatla
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: