Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-804

Cut down on the "mumbling" in the Task process' stdout/stderr

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.10.0
    • None
    • None

    Description

      Currently there is a lot of useless mumbling that Tasks do when running that needs to be removed. For example, here is a list of the output from a map:

      06/12/07 16:45:48 INFO mapred.TaskTracker: Child starting
      06/12/07 16:45:48 INFO conf.Configuration: parsing jar:file:/local/owen/hadoop-0.8.1-dev/hadoop-0.8.1-dev.jar!/hadoop-default.xml
      06/12/07 16:45:48 INFO conf.Configuration: parsing file:/local/owen/conf/mapred-default.xml
      06/12/07 16:45:48 INFO ipc.Client: org.apache.hadoop.io.ObjectWritableConnection culler maxidletime= 1000ms
      06/12/07 16:45:48 INFO ipc.Client: org.apache.hadoop.io.ObjectWritable Connection Culler: starting
      06/12/07 16:45:48 INFO conf.Configuration: parsing jar:file:/local/owen/hadoop-0.8.1-dev/hadoop-0.8.1-dev.jar!/hadoop-default.xml
      06/12/07 16:45:48 INFO conf.Configuration: parsing file:/local/owen/conf/mapred-default.xml
      06/12/07 16:45:48 INFO conf.Configuration: parsing /local/owen/run/tasktracker/taskTracker/jobcache/job_0002/task_0002_m_000000_0/job.xml
      06/12/07 16:45:49 INFO conf.Configuration: parsing jar:file:/local/owen/hadoop-0.8.1-dev/hadoop-0.8.1-dev.jar!/hadoop-default.xml
      06/12/07 16:45:49 INFO conf.Configuration: parsing file:/local/owen/conf/mapred-default.xml
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-0.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-1.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-2.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-3.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-4.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-5.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-6.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-7.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-8.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-9.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-10.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-11.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-12.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-13.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-14.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-15.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-16.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-17.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-18.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-19.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-20.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-21.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-22.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-23.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-24.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-25.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-26.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-27.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-28.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-29.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-30.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-31.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-32.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-33.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-34.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-35.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-36.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-37.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-38.out
      06/12/07 16:45:49 INFO mapred.MapTask: opened part-39.out

      Furthermore, it seems like the default logging level for Task subprocesses should be set to WARN.

      Attachments

        1. Hadoop-804.patch
          3 kB
          Sanjay Dahiya

        Issue Links

          Activity

            People

              sanjay.dahiya Sanjay Dahiya
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: