Zombie applications in the YARN queue using FAIR + sizebasedweight

Seeing zombie jobs in the YARN queue that uses FAIR and size based weight ordering policy .

Detection:

The YARN UI shows incorrect number of "Num Schedulable Applications".

Impact:

The queue has an upper limit of number of running applications, with zombie job, it hits the limit even though the number of running applications is far less than the limit. 

Workaround:

**Fail-over and restart Resource Manager process.

Analysis:

**In the heap dump, we can find the zombie jobs in the `FairOderingPolicy#

schedulableEntities` (see attachment). Take application "application_1599157165858_29429" for example, it is still in the  `FairOderingPolicy#schedulableEntities` set, however, if we check the log of resource manager, we can see RM already tried to remove the application:

./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 04:32:19,730 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Application removed - appId: application_1599157165858_29429 user: svc_di_data_eng queue: core-data #user-pending-applications: -3 #user-active-applications: 7 #queue-pending-applications: 0 #queue-active-applications: 21

So it appears RM failed to removed the application from the set.