Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-927

MapReduce is Broken for User-Defined Classes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 0.11.0
    • None
    • None
    • None

    Description

      Checked out trunk today and all of my jobs with a user-defined class as a map output value class fail with a ClassNotFoundException. Here's the stack trace:

      java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.StringWrapper
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
      at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:406)
      at org.apache.hadoop.mapred.ReduceTaskRunner.(ReduceTaskRunner.java:339)
      at org.apache.hadoop.mapred.ReduceTask.createRunner(ReduceTask.java:91)
      at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:983)
      at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:366)
      at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:359)
      at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:823)
      at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:510)
      at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:852)
      at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1494)

      I haven't looked deeply into it, but I would guess that the problem is in the constructor of ReduceTaskRunner where it attempts to create a SequenceFile.Sorter for the map output value class. It's probably done in the main TaskTracker JVM process, so external libraries are not on its classpath.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              albert.chern Albert Chern
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: