Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-2834

Make Tez preemption resilient to incorrect free resource reported by YARN

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.7.1
    • 0.7.1, 0.5.5, 0.6.3, 0.8.1-alpha
    • None
    • None
    • Reviewed

    Description

      Will attach the DAG.

      Repro for reference: TPC-DS q_70 @ 30 TB scale.

      "Map 7" completes in 2 waves. Output is very tiny, so reducer 8 gets launched slightly late. But before "Reducer 9" can get scheduled, slots are taken up by "Map 1", which is not preempted for running "Reducer 9".

      This is with 0.7.1 codebase.

      Attachments

        1. TEZ-2834.2.patch
          25 kB
          Bikas Saha
        2. TEZ-2834.1.patch
          11 kB
          Bikas Saha
        3. hive_view.png
          62 kB
          Rajesh Balamohan
        4. DAG_view.png
          103 kB
          Rajesh Balamohan
        5. application_1442254312093_0095.2.log.gz
          7.43 MB
          Rajesh Balamohan
        6. application_1442254312093_0095.1.log.gz
          6.68 MB
          Rajesh Balamohan

        Activity

          People

            bikassaha Bikas Saha
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: