Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
Reviewed
Description
Issues as part of Max apps in Capacity scheduler:
1. Cap total applications across the queue hierarchy based on existing max app calculation
2. Introduce a new configuration to take default max apps per queue irrespective of the queue capacity configuration
3. When the capacity configuration of the default partition is ZERO but queue has capacity for other partition then app is not getting submitted, though app is submitted in other partition
Steps to reproduce Issue 3 :
Configure capacity scheduler
yarn.scheduler.capacity.root.default.capacity=0
yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50
Submit application as below
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar sleep -Dmapreduce.job.node-label-expression=labelx -Dmapreduce.job.queuename=default -m 1 -r 1 -mt 10000000 -rt 1
2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001 java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1471670113386_0001 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1471670113386_0001 at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341) ... Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1471670113386_0001 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1471670113386_0001 at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286) at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301) ... 25 more