Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5220

Scheduling of OPPORTUNISTIC containers through YARN RM

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-beta1
    • resourcemanager
    • None
    • Hide
      This extends the centralized YARN RM in to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion.
      This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without needing to enable distributed scheduling.
      Show
      This extends the centralized YARN RM in to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion. This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without needing to enable distributed scheduling.

    Description

      In YARN-2882, we introduced the notion of OPPORTUNISTIC containers, along with the existing GUARANTEED containers of YARN.
      OPPORTUNISTIC containers are allowed to be queued at the NMs (YARN-2883), and are executed as long as there are available resources at the NM. Moreover, they are of lower priority than the GUARANTEED containers, that is, they can be preempted for a GUARANTEED container to start its execution.

      In YARN-2877, we introduced distributed scheduling in YARN, and enabled OPPORTUNISTIC containers to be scheduled exclusively by distributed schedulers.

      In this JIRA, we are proposing to extend the centralized YARN RM in order to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion.
      This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without the need to enable distributed scheduling.

      This JIRA is also related to YARN-1011 that introduces the over-commitment of resources, scheduling additional OPPORTUNISTIC containers to the NMs based on the currently used resources and not based only on the allocated resources.

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue YARN-5415 Add support for NodeLocal and RackLocal OPPORTUNISTIC requests

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue YARN-5688 Make allocation of opportunistic containers asynchronous

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. YARN-5823 Update NMTokens in case of requests with only opportunistic containers

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. YARN-5542 Scheduling of opportunistic containers

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. YARN-5646 Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          requires

          Sub-task - The sub-task of the issue YARN-2882 Add an OPPORTUNISTIC ExecutionType

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2883 Queuing of container requests in the NM

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              kkaranasos Konstantinos Karanasos
              kkaranasos Konstantinos Karanasos
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: