Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5139

[Umbrella] Move YARN scheduler towards global scheduler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions because scheduler can only look at one node at the time when scheduling resources.

      Pseudo code of existing scheduling logic looks like:

      for node in allNodes:
   Go to parentQueue
      Go to leafQueue
        for application in leafQueue.applications:
           for resource-request in application.resource-requests
              try to schedule on node

      Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

      Attachments

        1. YARN-5139.000.patch
          402 kB
          Wangda Tan
        2. wip-5.YARN-5139.patch
          386 kB
          Wangda Tan
        3. YARN-5139-Concurrent-scheduling-performance-report.pdf
          96 kB
          Wangda Tan
        4. Explanantions of Global Scheduling (YARN-5139) Implementation.pdf
          283 kB
          Wangda Tan
        5. wip-4.YARN-5139.patch
          447 kB
          Wangda Tan
        6. YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf
          285 kB
          Wangda Tan
        7. YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf
          245 kB
          Wangda Tan
        8. wip-3.YARN-5139.patch
          133 kB
          Wangda Tan
        9. wip-2.YARN-5139.patch
          95 kB
          Wangda Tan
        10. wip-1.YARN-5139.patch
          76 kB
          Wangda Tan

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. YARN-5181 ClusterNodeTracker: add method to get list of nodes matching a specific resourceName

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. YARN-5846 Improve the fairscheduler attemptScheduler

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-8575 Avoid committing allocation proposal to unavailable nodes in async scheduling

          • Major - Major loss of function.
          • Resolved
          is required by

          Sub-task - The sub-task of the issue YARN-1042 add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-4902 [Umbrella] Generalized and unified scheduling-strategies in YARN

          • Major - Major loss of function.
          • Resolved
          1.
          		 Add global scheduler interface definition and update CapacityScheduler to use it. Sub-task Resolved Wangda Tan  
          2.
          		 Update AppSchedulingInfo to use SchedulingPlacementSet Sub-task Resolved Wangda Tan  
          3.
          		 Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes Sub-task Resolved Wangda Tan  
          4.
          		 Update javadocs of new added APIs / classes of scheduler/AppSchedulingInfo Sub-task Open Wangda Tan  
          5.
          		 Should consider utilization of each ResourceType on node while scheduling Sub-task Open Qi Zhu  
          6.
          		 Global scheduler applies to Fair scheduler Sub-task Open Zhaohui Xin  
          7.
          		 Rename PlacementSet and SchedulingPlacementSet Sub-task Resolved Wangda Tan  
          8.
          		 Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm Sub-task Resolved Wangda Tan  
          9.
          		 Delay scheduling should be an individual policy instead of part of scheduler implementation Sub-task Open Tao Yang  
          10.
          		 Add multi-node lookup mechanism and pluggable nodes sorting policies to optimize placement decision Sub-task Resolved Sunil G  
          11.
          		 Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations Sub-task Resolved Sunil G  
          12.
          		 Pending backlog for async allocation threads should be configurable Sub-task Resolved Tao Yang  
          13.
          		 The capacity scheduler logs too frequently seriously affecting performance Sub-task Open YunFan Zhou  
          14.
          		 Resource leak caused by a reserved container being released more than once under async scheduling Sub-task Resolved Tao Yang  
          15.
          		 Support dynamic policy updates in Capacity Scheduler Sub-task Open Qi Zhu  
          16.
          		 Exclude lagged/unhealthy/decommissioned nodes in async allocating thread Sub-task Resolved Qi Zhu

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          17.
          		 Add multi-thread asynchronous scheduling to fair scheduler Sub-task Open Unassigned  
          18.
          		 Use threadPool to handle async scheduling threads Sub-task Open Aihua Xu  
          19.
          		 Skip schedule on not heartbeated nodes in Multi Node Placement Sub-task Resolved Prabhu Joseph  
          20.
          		 Proactively relocate allocated containers from a stopped node Sub-task Open Tanu Ajmera  
          21.
          		 Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement Sub-task Resolved Prabhu Joseph  
          22.
          		 Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement (YARN-10259) Sub-task Resolved Prabhu Joseph  
          23.
          		 Support Multi Node Placement in SingleConstraintAppPlacementAllocator Sub-task Resolved Prabhu Joseph  
          24.
          		 Import logic of multi-node allocation in CapacityScheduler Sub-task Resolved Qi Zhu

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 10m
          25.
          		 Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. Sub-task Resolved Qi Zhu  

          Activity

            People

              leftnoteasy Wangda Tan
              leftnoteasy Wangda Tan
              Votes:
              4 Vote for this issue
              Watchers:
              86 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h