PIG-2125-6.patch fixed many of core unit test failures for hadoop 23. There are still 20 failures left. I am working with Hadoop 23 team on these issues:
I am running Pig unit tests with hadoop 23. Here is some issues I current find:
1. Stability issue in MiniMRCluster. Some are intermittent, some fails consistently. Here is the stack I saw:
java.io.FileNotFoundException:
file:/home/daniel/pig/target/PigMiniCluster/PigMiniCluster-localDir/usercache/daniel/appcache/application_1313021517794_0001/container_1313021517794_0002_000001.tokens
at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:187)
at org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595)
at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:188)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:736)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:733)
at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2278)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:733)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:137)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:85)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:779)
If success, I didn't see it in the log
2. Couter issue. Seems most counter tests fail.
JobClient.getJob(job.getAssignedJobID()) always give me null.
3. MiniMRYarnCluster.getConfig() give me this entry:
"mapreduce.job.hdfs-servers=${fs.default.name}", which should have done the variable substitution.
4. Failure to start MiniHBaseCluster, all HBase tests fail.
Attach initial patch
PIG-2125-1.patch solving some issues we find. There are still areas under working, such as illustrator, MiniCluster, MergedJoin.