Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1197

Support changing resources of an allocated container

    XMLWordPrintableJSON

Details

    Description

      The current YARN resource management logic assumes resource allocated to a container is fixed during the lifetime of it. When users want to change a resource
      of an allocated container the only way is releasing it and allocating a new container with expected size.
      Allowing run-time changing resources of an allocated container will give us better control of resource usage in application side

      Attachments

        1. YARN-1197_Design.pdf
          692 kB
          MENG DING
        2. YARN-1197 old-design-docs-patches-for-reference.zip
          813 kB
          Wangda Tan
        3. YARN-1197_Design.2015.06.24.pdf
          685 kB
          MENG DING
        4. YARN-1197_Design.2015.07.07.pdf
          700 kB
          MENG DING
        5. YARN-1197_Design.2015.08.21.pdf
          692 kB
          MENG DING

        Issue Links

          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. SLIDER-938 Add ability to resize containers (Hadoop 2.8+)

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-896 Roll up for long-lived services in YARN

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-5085 Add support for change of container ExecutionType

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-4080 Capacity planning for long running services on YARN

          • Major - Major loss of function.
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-7180 Relaunching Failed Containers

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-3174 Provide elastic scaling within a Spark application

          • Major - Major loss of function.
          • Closed
          1.
          		 Common PB type definitions for container resizing Sub-task Closed Wangda Tan
          2.
          		 AM-RM protocol changes to support container resizing Sub-task Closed Wangda Tan
          3.
          		 AM-RM protocol changes to support container resizing Sub-task Resolved MENG DING
          4.
          		 AM-NM protocol changes to support container resizing Sub-task Resolved MENG DING
          5.
          		 ContainerManager implementation to support container resizing Sub-task Resolved MENG DING
          6.
          		 RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing Sub-task Resolved MENG DING
          7.
          		 ContainerImpl changes to support container resizing Sub-task Resolved MENG DING
          8.
          		 ContainerManager recovery for container resizing Sub-task Resolved MENG DING
          9.
          		 Make ContainersMonitor can support change monitoring size of an allocated container in NM side Sub-task Resolved MENG DING
          10.
          		 CapacityScheduler side changes to support increase/decrease container resource. Sub-task Resolved Wangda Tan
          11.
          		 Support change container resource in RM Sub-task Resolved Wangda Tan
          12.
          		 Roll back container resource allocation after resource increase token expires Sub-task Resolved MENG DING
          13.
          		 Make AMRMClient support send increase container request and get increased/decreased containers Sub-task Resolved MENG DING
          14.
          		 Make NMClient support change container resources Sub-task Resolved MENG DING
          15.
          		 Add implementations to FairScheduler to support increase/decrease container resource Sub-task Patch Available Wilfred Spiegelenburg
          16.
          		 Protocol changes in RM side to support change container resource Sub-task Resolved Unassigned
          17.
          		 [YARN-1197] Add increased/decreased container to Allocation Sub-task Resolved Unassigned
          18.
          		 [YARN-1197] Modify ApplicationMasterService to support changing container resource Sub-task Resolved Unassigned
          19.
          		 [YARN-1197] Modify ResourceTrackerService to support passing decreased containers to RMNode Sub-task Resolved Unassigned
          20.
          		 [YARN-1197] Add pullDecreasedContainer API to RMNode which can be used by scheduler to get newly decreased Containers Sub-task Resolved Unassigned
          21.
          		 [YARN-1197] Add methods in FiCaSchedulerNode to support increase/decrease/reserve/unreserve change container requests/results Sub-task Resolved Unassigned
          22.
          		 [YARN-1197] Add APIs in CSQueue to support decrease container resource and unreserve increase request Sub-task Resolved Unassigned
          23.
          		 [YARN-1197] Add implementations to CapacityScheduler to support increase/decrease container resource Sub-task Resolved Unassigned
          24.
          		 Merge YARN-1197 back to trunk Sub-task Resolved Wangda Tan
          25.
          		 Support changing container cpu resource Sub-task Resolved Yang Wang
          26.
          		 Resolve findbugs/javac warnings in YARN-1197 branch Sub-task Resolved Wangda Tan
          27.
          		 Example of use YARN-1197 Sub-task In Progress MENG DING
          28.
          		 Increasing container resource while there is no headroom left will cause ResourceManager to crash Sub-task Resolved MENG DING

          Activity

            People

              Unassigned Unassigned
              leftnoteasy Wangda Tan
              Votes:
              13 Vote for this issue
              Watchers:
              119 Start watching this issue

              Dates

                Created:
                Updated: