Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5139

[Umbrella] Move YARN scheduler towards global scheduler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions because scheduler can only look at one node at the time when scheduling resources.

      Pseudo code of existing scheduling logic looks like:

      for node in allNodes:
   Go to parentQueue
      Go to leafQueue
        for application in leafQueue.applications:
           for resource-request in application.resource-requests
              try to schedule on node

      Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

      Attachments

        1. YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf
          285 kB
          Wangda Tan
        2. YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf
          245 kB
          Wangda Tan
        3. YARN-5139-Concurrent-scheduling-performance-report.pdf
          96 kB
          Wangda Tan
        4. YARN-5139.000.patch
          402 kB
          Wangda Tan
        5. wip-5.YARN-5139.patch
          386 kB
          Wangda Tan
        6. wip-4.YARN-5139.patch
          447 kB
          Wangda Tan
        7. wip-3.YARN-5139.patch
          133 kB
          Wangda Tan
        8. wip-2.YARN-5139.patch
          95 kB
          Wangda Tan
        9. wip-1.YARN-5139.patch
          76 kB
          Wangda Tan
        10. Explanantions of Global Scheduling (YARN-5139) Implementation.pdf
          283 kB
          Wangda Tan

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. YARN-5181 ClusterNodeTracker: add method to get list of nodes matching a specific resourceName

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. YARN-5846 Improve the fairscheduler attemptScheduler

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-8575 Avoid committing allocation proposal to unavailable nodes in async scheduling

          • Major - Major loss of function.
          • Resolved
          is required by

          Sub-task - The sub-task of the issue YARN-1042 add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-4902 [Umbrella] Generalized and unified scheduling-strategies in YARN

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              leftnoteasy Wangda Tan
              leftnoteasy Wangda Tan
              Votes:
              4 Vote for this issue
              Watchers:
              86 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h