Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When I run TestPigRunner.classLoaderTest() in hadoop 23, I hit the following error:
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:281)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:233)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1211)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getInputPathFilter(FileInputFormat.java:182)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:214)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:269)
... 12 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1203)
... 17 more
Caused by: java.lang.ClassNotFoundException: org.apache.pig.test.PigTestLoader$TestPathFilter
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
... 18 more
The reason is when we register a jar, we do put the jar in the classloader of the current thread (PIG-1489). However, JobSubmitter is in a different thread and still use the default classloader. I am not sure if there is a way to change that classloader.
TestPigRunner.classLoaderTest() is actually wrong. The same test case will fail in mapreduce mode and 23 test.
Attachments
Issue Links
- is related to
-
PIG-2347 Fix Pig Unit tests for hadoop 23
- Closed