Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1198 Capacity Scheduler headroom calculation does not work as expected
  3. YARN-1680

availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.2.0, 2.3.0
    • None
    • capacityscheduler
    • None

    • SuSE 11 SP2 + Hadoop-2.3

    Description

      There are 4 NodeManagers with 8GB each.Total cluster capacity is 32GB.Cluster slow start is set to 1.

      Job is running reducer task occupied 29GB of cluster.One NodeManager(NM-4) is become unstable(3 Map got killed), MRAppMaster blacklisted unstable NodeManager(NM-4). All reducer task are running in cluster now.

      MRAppMaster does not preempt the reducers because for Reducer preemption calculation, headRoom is considering blacklisted nodes memory. This makes jobs to hang forever(ResourceManager does not assing any new containers on blacklisted nodes but returns availableResouce considers cluster free memory).

      Attachments

        1. YARN-1680-WIP.patch
          7 kB
          Chen He
        2. YARN-1680-v2.patch
          20 kB
          Chen He
        3. YARN-1680-v2.patch
          20 kB
          Chen He
        4. YARN-1680.patch
          18 kB
          Chen He

        Issue Links

          Is contained by

          Improvement - An improvement or enhancement to an existing feature or task. YARN-2848 (FICA) Applications should maintain an application specific 'cluster' resource to calculate headroom and userlimit

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5928 Deadlock allocating containers for mappers and reducers

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6302 Preempt reducers after a configurable timeout irrespective of headroom

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              Unassigned Unassigned
              rohithsharma Rohith Sharma K S
              Votes:
              1 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated: