Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11022

Fix the documentation for max-parallel-apps in CS

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      The documentation does not mention that the max-parallel-apps property is inherited. The property can be overridden on a per queue basis, but the parent(s) can also restrict how many parallel apps can be run.

       

      yarn.scheduler.capacity.max-parallel-apps / yarn.scheduler.capacity.<queue-path>.max-parallel-apps:

      Maximum number of applications that can run at the same time. Unlike to maximum-applications, application submissions are not rejected when this limit is reached. Instead they stay in ACCEPTED state until they are eligible to run. This can be set for all queues with yarn.scheduler.capacity.max-parallel-apps and can also be overridden on a per queue basis by setting yarn.scheduler.capacity.<queue-path>.max-parallel-apps. Integer value is expected. By default, there is no limit.

       

      https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java#L99

        private boolean exceedQueueMaxParallelApps(AbstractCSQueue queue) {
          // Check queue and all parent queues    while (queue != null) {
            if (queue.getNumRunnableApps() >= queue.getMaxParallelApps()) {
              LOG.info("Maximum runnable apps exceeded for queue {}",
                  queue.getQueuePath());
              return true;
            }
            queue = (AbstractCSQueue) queue.getParent();
          }    return false;
        } 
      

      Example:

      Let's say the user configured the yarn.scheduler.capacity.max-parallel-apps to 250, that will be the default for queues that doesn't override the setting. (https://github.com/apache/hadoop/blob/32ecaed9c3c06a48ef01d0437e62e8faccd3e9f3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1688)

      Given this queue hierarchy:

      root .a .a1 .a2 .a3 .a4
      500 default 50 10 default 15
      root .a .b
      500 default 50
      • maximum 250 apps can run parallel under the root.a queues.
      • maximum 50 apps can run parallel under the root.a.a1 queues.
      • maximum 10 apps can run parallel under the root.a.a1.a2 queues.
      • maximum 10 apps can run parallel under the root.a1.a2.a3 queues. (even though the max-parallel-apps is not set for .a3 so the default 250 applies for that queue, but it's parent had a lower value, and children can't exceed that)
      • maximum 10 apps can run parallel under the root.a1.a2.a3.a4 queue. (even though it's configured for 15, the parents restrict this limit to 10)
      • maximum 50 apps can run parallel under the root.a.b queue.

      Attachments

        Activity

          People

            tdomok Tamas Domok
            tdomok Tamas Domok
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 50m
                1h 50m