Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3808

NPE in FileOutputCommitter when running a 0 reduce job

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.23.0, 2.0.0-alpha
    • 0.23.1
    • mrv2
    • None
    • Reviewed
    • Fixed an NPE in FileOutputCommitter for jobs with maps but no reduces.

    Description

      This was while running LoadGen.

      Error: java.lang.NullPointerException at org.apache.hadoop.fs.Path.<init>(Path.java:67) 
at org.apache.hadoop.fs.Path.<init>(Path.java:56) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209) 
at org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100) 
at org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94) 
at org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176) 
at org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248) 
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955) 
at org.apache.hadoop.mapred.Task.done(Task.java:912) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:396) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)

      Attachments

        1. MR-3808.txt
          9 kB
          Robert Joseph Evans

        Issue Links

          blocks

          Sub-task - The sub-task of the issue MAPREDUCE-3524 Scan benchmark is more than 1.5x slower in 0.23

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-3821 NPE while running Shuffle benchmark

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is broken by

          Sub-task - The sub-task of the issue MAPREDUCE-3711 AppMaster recovery for Medium to large jobs take long time

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              revans2 Robert Joseph Evans
              sseth Siddharth Seth
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: