Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-299

maps from second jobs will not run until the first job finishes completely

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.3.2
    • 0.4.0
    • None
    • None

    Description

      Because of the logic in the JobTracker's pollForNewTask, second jobs will rarely start running maps until the first job finishes completely. The JobTracker leaves room to re-run failed maps from the first job and it reserves the total number of maps for the first job. Thus, if you have more maps in the first job than your cluster capacity, none of the second job maps will ever run.

      I propose setting the reserve to 1% of the first job's maps.

      Attachments

        1. map-schedule.patch
          4 kB
          Owen O'Malley

        Issue Links

          Activity

            People

              omalley Owen O'Malley
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: