Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3107

DFSIO tests are failing intermittently

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Cannot Reproduce
    • 0.23.0
    • 0.23.0
    • benchmarks, mrv2
    • None

    Description

      Intermittently DFSIO tests are failing either in "read" or "write" operations.

      $HADOOP_COMMON_HOME/bin/hadoop -config $HADOOP_CONF_DIR jar $HADOOP_MAPRED_HOME/hadoop-mapreduce-test*.jar TestDFSIO -read -nrFiles 36 -fileSize 320

      11/09/27 03:21:03 INFO fs.TestDFSIO: TestDFSIO.0.0.6
      11/09/27 03:21:03 INFO fs.TestDFSIO: nrFiles = 36
      11/09/27 03:21:03 INFO fs.TestDFSIO: fileSize (MB) = 320.0
      11/09/27 03:21:03 INFO fs.TestDFSIO: bufferSize = 1000000
      11/09/27 03:21:03 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
      11/09/27 03:21:03 INFO fs.TestDFSIO: creating control file: 335544320 bytes, 36 files
      11/09/27 03:21:05 INFO fs.TestDFSIO: created control files for: 36 files
      ..
      ..
      ..
      ..
      ..
      11/09/27 03:24:04 INFO mapreduce.Job: map 88% reduce 25%
      11/09/27 03:24:05 INFO mapreduce.Job: map 90% reduce 25%
      11/09/27 03:24:06 INFO mapreduce.Job: map 92% reduce 25%
      11/09/27 03:24:09 INFO mapreduce.Job: map 93% reduce 25%
      11/09/27 03:24:10 INFO mapreduce.Job: map 93% reduce 26%
      11/09/27 03:24:42 INFO mapreduce.Job: map 94% reduce 26%
      11/09/27 03:24:43 INFO mapreduce.Job: map 94% reduce 27%
      11/09/27 03:24:56 INFO mapreduce.Job: map 95% reduce 27%
      11/09/27 03:24:58 INFO mapreduce.Job: map 95% reduce 28%
      11/09/27 03:25:13 INFO mapreduce.Job: map 96% reduce 28%
      11/09/27 03:25:15 INFO mapreduce.Job: map 97% reduce 28%
      11/09/27 03:25:16 INFO mapreduce.Job: map 97% reduce 30%
      11/09/27 03:25:16 INFO mapreduce.Job: map 98% reduce 30%
      11/09/27 03:25:16 INFO mapreduce.Job: Job job_1317092846056_0012 failed with state FAILED
      11/09/27 03:25:16 INFO mapreduce.Job: Counters: 44
      File System Counters
      FILE: BYTES_READ=20931
      FILE: BYTES_WRITTEN=2192966
      FILE: READ_OPS=0
      FILE: LARGE_READ_OPS=0
      FILE: WRITE_OPS=0
      HDFS: BYTES_READ=10454250228
      HDFS: BYTES_WRITTEN=0
      HDFS: READ_OPS=172
      HDFS: LARGE_READ_OPS=0
      HDFS: WRITE_OPS=0
      org.apache.hadoop.mapreduce.JobCounter
      NUM_FAILED_MAPS=6
      TOTAL_LAUNCHED_MAPS=44
      TOTAL_LAUNCHED_REDUCES=1
      DATA_LOCAL_MAPS=10
      RACK_LOCAL_MAPS=34
      SLOTS_MILLIS_MAPS=2340307
      org.apache.hadoop.mapreduce.TaskCounter
      MAP_INPUT_RECORDS=36
      MAP_OUTPUT_RECORDS=140
      MAP_OUTPUT_BYTES=2103
      MAP_OUTPUT_MATERIALIZED_BYTES=2579
      SPLIT_RAW_BYTES=5210
      COMBINE_INPUT_RECORDS=0
      COMBINE_OUTPUT_RECORDS=0
      REDUCE_INPUT_GROUPS=0
      REDUCE_SHUFFLE_BYTES=3054
      REDUCE_INPUT_RECORDS=0
      REDUCE_OUTPUT_RECORDS=0
      SPILLED_RECORDS=140
      SHUFFLED_MAPS=33
      FAILED_SHUFFLE=0
      MERGED_MAP_OUTPUTS=0
      GC_TIME_MILLIS=59933
      CPU_MILLISECONDS=159470
      PHYSICAL_MEMORY_BYTES=11310596096
      VIRTUAL_MEMORY_BYTES=31425290240
      COMMITTED_HEAP_BYTES=12728664064
      Shuffle Errors
      BAD_ID=0
      CONNECTION=0
      IO_ERROR=0
      WRONG_LENGTH=0
      WRONG_MAP=0
      WRONG_REDUCE=0
      org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
      BYTES_READ=4058
      org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter
      BYTES_WRITTEN=0
      java.io.IOException: Job failed!
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:781)
      at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:340)
      at org.apache.hadoop.fs.TestDFSIO.readTest(TestDFSIO.java:418)
      at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:522)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
      at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:445)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
      at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
      at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:189)

      Attachments

        Activity

          People

            johnvijoe John George
            rajsaha Rajit Saha
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: