Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5583

Ability to limit running map and reduce tasks

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.9, 2.1.1-beta
    • 2.7.0
    • mr-am, mrv2
    • None
    • Reviewed
    • Hide
      <!-- markdown -->
      This introduces two new MR2 job configs, mentioned below, which allow users to control the maximum simultaneously-running tasks of the submitted job, across the cluster:

      * mapreduce.job.running.map.limit (default: 0, for no limit)
      * mapreduce.job.running.reduce.limit (default: 0, for no limit)

      This is controllable at a per-job level.
      Show
      <!-- markdown --> This introduces two new MR2 job configs, mentioned below, which allow users to control the maximum simultaneously-running tasks of the submitted job, across the cluster: * mapreduce.job.running.map.limit (default: 0, for no limit) * mapreduce.job.running.reduce.limit (default: 0, for no limit) This is controllable at a per-job level.

    Description

      It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time.

      This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2.

      Attachments

        1. MAPREDUCE-5583v4.patch
          28 kB
          Jason Darrell Lowe
        2. MAPREDUCE-5583v3.patch
          28 kB
          Jason Darrell Lowe
        3. MAPREDUCE-5583v2.patch
          28 kB
          Jason Darrell Lowe
        4. MAPREDUCE-5583v1.patch
          17 kB
          Jason Darrell Lowe
        5. MAPREDUCE-5583-branch2.4.1.patch
          16 kB
          Yang Hao

        Issue Links

          is duplicated by

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6176 To limit the map task number or reduce task number of an application

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-3274 FIFOScheduler should assign the same container number up to ResourceRequest ask for ANY

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6697 Concurrent task limits should only be applied when necessary

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-7274 Enable to limit running map and reduce tasks when job is running

          • Major - Major loss of function.
          • Patch Available
          relates to

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-224 limit running tasks per job

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-5170 Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jlowe Jason Darrell Lowe
              jlowe Jason Darrell Lowe
              Votes:
              2 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: