Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10634

The config parameter "mapreduce.job.num-opportunistic-maps-percent" is confusing when requesting Opportunistic containers in YARN job

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • applications
    • None

    Description

      Execute the below job by Passing this config -Dmapreduce.job.num-opportunistic-maps-percent ,which actually represents the number of containers to be launched as Opportunistic, not in % of the total mappers requested , i think this configuration name should be modified accordingly and also the same gets printed in AM logs which also needs to be corrected accordingly.

      Job Command: hadoop jar HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1-hw-ei-310001-SNAPSHOT.jar pi -Dmapreduce.job.num-opportunistic-maps-percent="20" 20 99

      In AM logs this message is displayed. it should be 20 , not 20% ?
      “2021-02-10 20:23:23,023 | INFO | main | 20% of the mappers will be scheduled using OPPORTUNISTIC containers | RMContainerAllocator.java:257”

      Job Command: hadoop jar HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1-hw-ei-310001-SNAPSHOT.jar pi -Dmapreduce.job.num-opportunistic-maps-percent="100" 20 99

      In AM logs this message is displayed. It should be 100, not 100% ?

      2021-02-10 20:28:16,016 | INFO  | main | 100% of the mappers will be scheduled using OPPORTUNISTIC containers | RMContainerAllocator.java:257

      Attachments

        Activity

          People

            BilwaST Bilwa S T
            koolsen@gmail.com Sushanta Sen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: