Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2449

register statement fail to change classloader in JobSubmitter thread

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • impl
    • None

    Description

      When I run TestPigRunner.classLoaderTest() in hadoop 23, I hit the following error:
      org.apache.pig.backend.executionengine.ExecException: ERROR 2118: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:281)
      at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445)
      at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462)
      at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
      at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
      at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
      at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
      at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
      at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:233)
      at java.lang.Thread.run(Thread.java:680)
      Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1211)
      at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getInputPathFilter(FileInputFormat.java:182)
      at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:214)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
      at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:269)
      ... 12 more
      Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1203)
      ... 17 more
      Caused by: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:247)
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
      ... 18 more

      The reason is when we register a jar, we do put the jar in the classloader of the current thread (PIG-1489). However, JobSubmitter is in a different thread and still use the default classloader. I am not sure if there is a way to change that classloader.

      TestPigRunner.classLoaderTest() is actually wrong. The same test case will fail in mapreduce mode and 23 test.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              daijy Daniel Dai
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: