Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2119

JobTracker becomes non-responsive if the task trackers finish task too fast

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.16.0
    • 0.17.0
    • None
    • None
    • Reviewed
    • Hide
      This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.
      Show
      This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.

    Description

      I ran a job with 0 reducer on a cluster with 390 nodes.
      The mappers ran very fast.
      The jobtracker lacks behind on committing completed mapper tasks.
      The number of running mappers displayed on web UI getting bigger and bigger.
      The jos tracker eventually stopped responding to web UI.

      No progress is reported afterwards.

      Job tracker is running on a separate node.
      The job tracker process consumed 100% cpu, with vm size 1.01g (reach the heap space limit).

      Attachments

        1. hadoop-jobtracker-thread-dump.txt
          60 kB
          Christian Kunz
        2. hadoop-2119.patch
          11 kB
          Srikanth Kakani
        3. HADOOP-2119-v4.1.patch
          52 kB
          Amar Kamat
        4. HADOOP-2119-v5.1.patch
          52 kB
          Amar Kamat
        5. HADOOP-2119-v5.1.patch
          55 kB
          Amar Kamat
        6. HADOOP-2119-v5.2.patch
          51 kB
          Amar Kamat

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-315 Bias the decision of task scheduling (both for not-running and running) on node metrics (load, processing rate etc).

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-262 Optimize finding of speculative tasks

          • Major - Major loss of function.
          • Resolved
          depends upon

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-1985 Abstract node to switch mapping into a topology service class used by namenode and jobtracker

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-93 Job Tracker should prefer input-splits from overloaded racks

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-2790 TaskInProgress.hasSpeculativeTask is very inefficient

          • Major - Major loss of function.
          • Closed

          Activity

            People

              amar_kamat Amar Kamat
              runping Runping Qi
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: