Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8737

Race condition in ParentQueue when reinitializing and sorting child queues in the meanwhile

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 3.3.0, 2.9.3, 3.2.2, 3.1.4
    • None
    • capacityscheduler
    • None

    Description

      Administrator raised a update for queues through REST API, in RM parent queue is refreshing child queues through calling ParentQueue#reinitialize, meanwhile, async-schedule threads is sorting child queues when calling ParentQueue#sortAndGetChildrenAllocationIterator. Race condition may happen and throw exception as follow because TimSort does not handle the concurrent modification of objects it is sorting:

      java.lang.IllegalArgumentException: Comparison method violates its general contract!
        at java.util.TimSort.mergeHi(TimSort.java:899)
        at java.util.TimSort.mergeAt(TimSort.java:516)
        at java.util.TimSort.mergeCollapse(TimSort.java:441)
        at java.util.TimSort.sort(TimSort.java:245)
        at java.util.Arrays.sort(Arrays.java:1512)
        at java.util.ArrayList.sort(ArrayList.java:1454)
        at java.util.Collections.sort(Collections.java:175)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:291)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:804)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:817)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:636)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:2494)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:2431)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersOnMultiNodes(CapacityScheduler.java:2588)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:2676)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.scheduleBasedOnNodeLabels(CapacityScheduler.java:927)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:962)

      I think we can add read-lock for ParentQueue#sortAndGetChildrenAllocationIterator to solve this problem, the write-lock will be hold when updating child queues in ParentQueue#reinitialize.

      Attachments

        1. YARN-8737.001.patch
          1 kB
          Tao Yang

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. YARN-10058 Capacity Scheduler dispatcher hang when async thread crash

          • Major - Major loss of function.
          • Patch Available
          is fixed by

          Bug - A problem which impairs or prevents the functions of the product. YARN-10178 Global Scheduler async thread crash caused by 'Comparison method violates its general contract

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Tao Yang Tao Yang
              Tao Yang Tao Yang
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: