Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2125

Make Pig work with hadoop .NEXT

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.9.1, 0.10.0
    • 0.9.2, 0.10.0
    • impl
    • None
    • Reviewed

    Description

      We need to make Pig work with hadoop .NEXT, the svn branch currently is: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279

      Attachments

        1. PIG-2125-zebra.patch
          2 kB
          Daniel Dai
        2. PIG-2125-ivy-0.9-3.patch
          4 kB
          Thomas Weise
        3. PIG-2125-commitJob.patch
          3 kB
          Daniel Dai
        4. PIG-2125-buildxml-0.9.patch
          3 kB
          Thomas Weise
        5. PIG-2125-9.patch
          84 kB
          Daniel Dai
        6. PIG-2125-9_0.9.patch
          83 kB
          Daniel Dai
        7. PIG-2125-8.patch
          84 kB
          Daniel Dai
        8. PIG-2125-7.patch
          74 kB
          Daniel Dai
        9. PIG-2125-7_0.9.patch
          72 kB
          Daniel Dai
        10. PIG-2125-6.patch
          72 kB
          Daniel Dai
        11. PIG-2125-5.patch
          3 kB
          Daniel Dai
        12. PIG-2125-4.patch
          183 kB
          Daniel Dai
        13. PIG-2125-3.patch
          0.8 kB
          Daniel Dai
        14. PIG-2125-2.patch
          169 kB
          Daniel Dai
        15. PIG-2125-10.patch
          84 kB
          Daniel Dai
        16. PIG-2125-10_0.9.patch
          84 kB
          Daniel Dai
        17. PIG-2125-1.patch
          20 kB
          Daniel Dai
        18. e2e-hadoop23.patch
          0.5 kB
          Daniel Dai
        19. ContextFactory.java
          9 kB
          Daniel Dai

        Issue Links

          Activity

            daijy Daniel Dai added a comment -

            Attach initial patch PIG-2125-1.patch solving some issues we find. There are still areas under working, such as illustrator, MiniCluster, MergedJoin.

            daijy Daniel Dai added a comment - Attach initial patch PIG-2125 -1.patch solving some issues we find. There are still areas under working, such as illustrator, MiniCluster, MergedJoin.

            are you guys planning on skipping 22 altogether?

            dvryaboy Dmitriy V. Ryaboy added a comment - are you guys planning on skipping 22 altogether?
            olgan Olga Natkovich added a comment -

            We don't have any current plans to use 22 at Yahoo or for Yahoo engineers to do the port. But of course a patch is welcome from anybody interested in taking this project on.

            olgan Olga Natkovich added a comment - We don't have any current plans to use 22 at Yahoo or for Yahoo engineers to do the port. But of course a patch is welcome from anybody interested in taking this project on.
            daijy Daniel Dai added a comment -

            PIG-2125-2.patch tested against MR-279 branch. All Pig end-to-end test pass. Unit test still fail. Zebra test still have a few failures.

            The patch borrow some ideas from PIG-924. However, a dynamic shims layer is not possible due to some static dependencies. Here we use a static shims layer, so for different hadoop version, we need to recompile.

            Since hadoop.Next maven artifacts is not available, so we need to simulate it by copying all libraries to lib.

            daijy Daniel Dai added a comment - PIG-2125 -2.patch tested against MR-279 branch. All Pig end-to-end test pass. Unit test still fail. Zebra test still have a few failures. The patch borrow some ideas from PIG-924 . However, a dynamic shims layer is not possible due to some static dependencies. Here we use a static shims layer, so for different hadoop version, we need to recompile. Since hadoop.Next maven artifacts is not available, so we need to simulate it by copying all libraries to lib.
            daijy Daniel Dai added a comment -

            PIG-2125-3.patch is additional patch to make Zebra work.

            daijy Daniel Dai added a comment - PIG-2125 -3.patch is additional patch to make Zebra work.
            gates Alan Gates added a comment -

            In general the patch looks good. I have a few questions/comments.

            1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA.
            2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed.
            3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.
            gates Alan Gates added a comment - In general the patch looks good. I have a few questions/comments. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.
            daijy Daniel Dai added a comment -

            1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA.

            They need to copy all hadoop 23 jars and dependencies into lib. This step should go away once hadoop 23 publish maven artifacts

            2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed.

            Sure. I will add

            3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.

            How about org.apache.pig.backend.hadoop.shims?

            daijy Daniel Dai added a comment - 1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA. They need to copy all hadoop 23 jars and dependencies into lib. This step should go away once hadoop 23 publish maven artifacts 2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed. Sure. I will add 3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better. How about org.apache.pig.backend.hadoop.shims?
            daijy Daniel Dai added a comment -

            PIG-2125-4.patch merge PIG-2125-2.patch and PIG-2125-3.PATCH, address comments from Alan, fix unit test failures and test-patch failures.

            daijy Daniel Dai added a comment - PIG-2125 -4.patch merge PIG-2125 -2.patch and PIG-2125 -3.PATCH, address comments from Alan, fix unit test failures and test-patch failures.
            daijy Daniel Dai added a comment -

            PIG-2125-4.patch committed to both trunk and 0.9 branch. This is the first phase work (mapreduce and local mode end-to-end), I will continue work on the next phase (unit test).

            daijy Daniel Dai added a comment - PIG-2125 -4.patch committed to both trunk and 0.9 branch. This is the first phase work (mapreduce and local mode end-to-end), I will continue work on the next phase (unit test).
            daijy Daniel Dai added a comment -

            There are some piggybank and zebra test failures, attach PIG-2125-5.patch for the fix.

            daijy Daniel Dai added a comment - There are some piggybank and zebra test failures, attach PIG-2125 -5.patch for the fix.
            rding Richard Ding added a comment -

            +1

            rding Richard Ding added a comment - +1
            daijy Daniel Dai added a comment -

            PIG-2125-5.patch committed to both trunk and 0.9 branch.

            daijy Daniel Dai added a comment - PIG-2125 -5.patch committed to both trunk and 0.9 branch.
            daijy Daniel Dai added a comment -

            Steps to build pig.jar for hadoop 23:
            1. change "build.xml", modify "hadoopversion" to "23"
            2. change "ivy.xml", remove the lines:
            <dependency org="org.apache.hadoop" name="hadoop-core" rev="${hadoop-core.version}"
            conf="compile->default;test->default"/>
            <dependency org="org.apache.hadoop" name="hadoop-test" rev="${hadoop-test.version}"
            conf="compile->default;test->default"/>
            3. Build hadoop 23, copy hadoop 23 libraries into lib, here is a list:
            aopalliance-1.0.jar
            commons-cli-1.2.jar
            commons-codec-1.4.jar
            commons-collections-3.2.1.jar
            commons-configuration-1.6.jar
            commons-httpclient-3.1.jar
            guice-2.0.jar
            guice-servlet-2.0.jar
            hadoop-common-0.22.0-SNAPSHOT.jar
            hadoop-common-test-0.22.0-SNAPSHOT.jar
            hadoop-hdfs-0.22.0-SNAPSHOT.jar
            hadoop-hdfs-test-0.22.0-SNAPSHOT.jar
            hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar
            hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar
            hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar
            hadoop-mapreduce-client-hs-1.0-SNAPSHOT.jar
            hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
            hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT-tests.jar
            hadoop-mapreduce-client-shuffle-1.0-SNAPSHOT.jar
            hadoop-metrics2-0.22.0-SNAPSHOT.jar
            jetty-6.1.26.jar
            log4j-1.2.16.jar
            protobuf-java-2.4.0a.jar
            servlet-api-2.5-20081211.jar
            slf4j-api-1.6.1.jar
            slf4j-log4j12-1.6.1.jar
            yarn-api-1.0-SNAPSHOT.jar
            yarn-common-1.0-SNAPSHOT.jar
            yarn-server-common-1.0-SNAPSHOT.jar
            yarn-server-nodemanager-1.0-SNAPSHOT.jar
            yarn-server-resourcemanager-1.0-SNAPSHOT.jar
            yarn-server-tests-1.0-SNAPSHOT-tests.jar

            4. Then, do an "ant clean jar"
            5. You may also need to recompile your udf with the new hadoop libraries.

            daijy Daniel Dai added a comment - Steps to build pig.jar for hadoop 23: 1. change "build.xml", modify "hadoopversion" to "23" 2. change "ivy.xml", remove the lines: <dependency org="org.apache.hadoop" name="hadoop-core" rev="${hadoop-core.version}" conf="compile->default;test->default"/> <dependency org="org.apache.hadoop" name="hadoop-test" rev="${hadoop-test.version}" conf="compile->default;test->default"/> 3. Build hadoop 23, copy hadoop 23 libraries into lib, here is a list: aopalliance-1.0.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.1.jar commons-configuration-1.6.jar commons-httpclient-3.1.jar guice-2.0.jar guice-servlet-2.0.jar hadoop-common-0.22.0-SNAPSHOT.jar hadoop-common-test-0.22.0-SNAPSHOT.jar hadoop-hdfs-0.22.0-SNAPSHOT.jar hadoop-hdfs-test-0.22.0-SNAPSHOT.jar hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar hadoop-mapreduce-client-hs-1.0-SNAPSHOT.jar hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT-tests.jar hadoop-mapreduce-client-shuffle-1.0-SNAPSHOT.jar hadoop-metrics2-0.22.0-SNAPSHOT.jar jetty-6.1.26.jar log4j-1.2.16.jar protobuf-java-2.4.0a.jar servlet-api-2.5-20081211.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar yarn-api-1.0-SNAPSHOT.jar yarn-common-1.0-SNAPSHOT.jar yarn-server-common-1.0-SNAPSHOT.jar yarn-server-nodemanager-1.0-SNAPSHOT.jar yarn-server-resourcemanager-1.0-SNAPSHOT.jar yarn-server-tests-1.0-SNAPSHOT-tests.jar 4. Then, do an "ant clean jar" 5. You may also need to recompile your udf with the new hadoop libraries.
            daijy Daniel Dai added a comment -

            Attach ContextFactory.java which eventually will be in hadoop's codebase. But for now, we need to put it in Pig's source tree in order to compile.

            daijy Daniel Dai added a comment - Attach ContextFactory.java which eventually will be in hadoop's codebase. But for now, we need to put it in Pig's source tree in order to compile.
            daijy Daniel Dai added a comment -

            Put ContextFactory.java into $PIG_HOME/src/org/apache/hadoop/mapreduce, then compile.

            daijy Daniel Dai added a comment - Put ContextFactory.java into $PIG_HOME/src/org/apache/hadoop/mapreduce, then compile.
            daijy Daniel Dai added a comment -

            PIG-2125-6.patch fixed many of core unit test failures for hadoop 23. There are still 20 failures left. I am working with Hadoop 23 team on these issues:
            I am running Pig unit tests with hadoop 23. Here is some issues I current find:

            1. Stability issue in MiniMRCluster. Some are intermittent, some fails consistently. Here is the stack I saw:
            java.io.FileNotFoundException:
            file:/home/daniel/pig/target/PigMiniCluster/PigMiniCluster-localDir/usercache/daniel/appcache/application_1313021517794_0001/container_1313021517794_0002_000001.tokens
            at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:187)
            at org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150)
            at org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595)
            at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:188)
            at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:736)
            at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:733)
            at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2278)
            at org.apache.hadoop.fs.FileContext.open(FileContext.java:733)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:137)
            at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:85)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:779)

            If success, I didn't see it in the log

            2. Couter issue. Seems most counter tests fail.
            JobClient.getJob(job.getAssignedJobID()) always give me null.

            3. MiniMRYarnCluster.getConfig() give me this entry:
            "mapreduce.job.hdfs-servers=${fs.default.name}", which should have done the variable substitution.

            4. Failure to start MiniHBaseCluster, all HBase tests fail.

            daijy Daniel Dai added a comment - PIG-2125 -6.patch fixed many of core unit test failures for hadoop 23. There are still 20 failures left. I am working with Hadoop 23 team on these issues: I am running Pig unit tests with hadoop 23. Here is some issues I current find: 1. Stability issue in MiniMRCluster. Some are intermittent, some fails consistently. Here is the stack I saw: java.io.FileNotFoundException: file:/home/daniel/pig/target/PigMiniCluster/PigMiniCluster-localDir/usercache/daniel/appcache/application_1313021517794_0001/container_1313021517794_0002_000001.tokens at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:187) at org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150) at org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595) at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:188) at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:736) at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:733) at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2278) at org.apache.hadoop.fs.FileContext.open(FileContext.java:733) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:137) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:85) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:779) If success, I didn't see it in the log 2. Couter issue. Seems most counter tests fail. JobClient.getJob(job.getAssignedJobID()) always give me null. 3. MiniMRYarnCluster.getConfig() give me this entry: "mapreduce.job.hdfs-servers=${fs.default.name}", which should have done the variable substitution. 4. Failure to start MiniHBaseCluster, all HBase tests fail.
            daijy Daniel Dai added a comment -

            PIG-2125-7.patch catch up with Pig trunk, and compilable against Hadoop 0.23 brach.

            daijy Daniel Dai added a comment - PIG-2125 -7.patch catch up with Pig trunk, and compilable against Hadoop 0.23 brach.
            daijy Daniel Dai added a comment -

            e2e-hadoop23.patch is a separable patch from PIG-2125-7.patch which enable e2e test only.

            daijy Daniel Dai added a comment - e2e-hadoop23.patch is a separable patch from PIG-2125 -7.patch which enable e2e test only.
            daijy Daniel Dai added a comment -

            Missing one file in PIG-2125-7.patch, reattach.

            daijy Daniel Dai added a comment - Missing one file in PIG-2125 -7.patch, reattach.
            daijy Daniel Dai added a comment -

            The same patch for 0.9 branch.

            daijy Daniel Dai added a comment - The same patch for 0.9 branch.
            acmurthy Arun Murthy added a comment -

            I'll get ContextFactory committed to branch-0.23 so that Pig doesn't need to own this.

            acmurthy Arun Murthy added a comment - I'll get ContextFactory committed to branch-0.23 so that Pig doesn't need to own this.
            acmurthy Arun Murthy added a comment -

            I've committed MAPREDUCE-3138 i.e. the ContextFactory patch to Hadoop MapReduce. Thanks.

            acmurthy Arun Murthy added a comment - I've committed MAPREDUCE-3138 i.e. the ContextFactory patch to Hadoop MapReduce. Thanks.
            thw Thomas Weise added a comment -

            This patch is to support hadoopversion=23 to resolve dependencies through maven.

            Note that currently mapreduce does not have the jar files in maven, so you will still need to copy those into lib/

            thw Thomas Weise added a comment - This patch is to support hadoopversion=23 to resolve dependencies through maven. Note that currently mapreduce does not have the jar files in maven, so you will still need to copy those into lib/
            thw Thomas Weise added a comment -

            PIG-2125-buildxml-0.9.patch - Changes to build.xml for buildJar target. Verified changes for .20 build via diff of jar tv ... before and after.

            File list for .23 fat jar needs to be tuned once all maven dependencies are in place and tests compile/pass.

            thw Thomas Weise added a comment - PIG-2125 -buildxml-0.9.patch - Changes to build.xml for buildJar target. Verified changes for .20 build via diff of jar tv ... before and after. File list for .23 fat jar needs to be tuned once all maven dependencies are in place and tests compile/pass.
            daijy Daniel Dai added a comment -

            PIG-2125-commitJob.patch is to address incompatibility change introduced in MAPREDUCE-2702

            daijy Daniel Dai added a comment - PIG-2125 -commitJob.patch is to address incompatibility change introduced in MAPREDUCE-2702
            daijy Daniel Dai added a comment -

            Tests pass for PIG-2125-commitJob.patch. No test included since it solve the problem only in hadoop 23, which is not released.

            daijy Daniel Dai added a comment - Tests pass for PIG-2125 -commitJob.patch. No test included since it solve the problem only in hadoop 23, which is not released.
            olgan Olga Natkovich added a comment -

            Did the tests pass for both .Next and 20.x version?

            olgan Olga Natkovich added a comment - Did the tests pass for both .Next and 20.x version?
            daijy Daniel Dai added a comment -

            All tests done on 20.X. I only did sanity tests on .Next, since I don't have the facility to run the whole thing currently.

            daijy Daniel Dai added a comment - All tests done on 20.X. I only did sanity tests on .Next, since I don't have the facility to run the whole thing currently.
            olgan Olga Natkovich added a comment -

            I think that is good enough. As long as we do not regress on the current Hadoop I think we are good. Just need a review then.

            olgan Olga Natkovich added a comment - I think that is good enough. As long as we do not regress on the current Hadoop I think we are good. Just need a review then.
            thejas Thejas Nair added a comment -

            +1 for PIG-2125-commitJob.patch

            thejas Thejas Nair added a comment - +1 for PIG-2125 -commitJob.patch
            daijy Daniel Dai added a comment -

            PIG-2125-zebra.patch fix zebra test failures introduced by PIG-2125-commitJob.patch

            daijy Daniel Dai added a comment - PIG-2125 -zebra.patch fix zebra test failures introduced by PIG-2125 -commitJob.patch
            thw Thomas Weise added a comment -

            Update to the mapreduce dependencies (replaces previous patch).

            thw Thomas Weise added a comment - Update to the mapreduce dependencies (replaces previous patch).
            thw Thomas Weise added a comment -

            Modified ivy patch to pull compile-test dependencies correctly. Currently the snapshots are outdated, compile-test should pass once they are refreshed.

            thw Thomas Weise added a comment - Modified ivy patch to pull compile-test dependencies correctly. Currently the snapshots are outdated, compile-test should pass once they are refreshed.
            daijy Daniel Dai added a comment -

            PIG-2125-8.patch makes Pig compile and run against both 23 and 20 using maven artifacts. The only exception is hadoop-mapreduce.jar, which hadoop still fail to publish, and we need to copy it into lib. PIG-2125-8.patch include all changes in PIG-2125-7.patch, PIG-2125-buildxml-0.9.patch and PIG-2125-ivy-0.9-3.patch.

            daijy Daniel Dai added a comment - PIG-2125 -8.patch makes Pig compile and run against both 23 and 20 using maven artifacts. The only exception is hadoop-mapreduce.jar, which hadoop still fail to publish, and we need to copy it into lib. PIG-2125 -8.patch include all changes in PIG-2125 -7.patch, PIG-2125 -buildxml-0.9.patch and PIG-2125 -ivy-0.9-3.patch.
            daijy Daniel Dai added a comment -

            PIG-2125-9.patch & PIG-2125-9_0.9.patch compile and run Pig without any manual intervention.

            daijy Daniel Dai added a comment - PIG-2125 -9.patch & PIG-2125 -9_0.9.patch compile and run Pig without any manual intervention.
            thw Thomas Weise added a comment -

            +1 ant compile-test is good on branch-0.9 now

            thw Thomas Weise added a comment - +1 ant compile-test is good on branch-0.9 now
            daijy Daniel Dai added a comment -

            PIG-2125-10.patch, PIG-2125-10_0.9.patch fix test-patch warnings:
            [exec] -1 overall.
            [exec]
            [exec] +1 @author. The patch does not contain any @author tags.
            [exec]
            [exec] +1 tests included. The patch appears to include 46 new or modified tests.
            [exec]
            [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
            [exec]
            [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
            [exec]
            [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
            [exec]
            [exec] -1 release audit. The applied patch generated 462 release audit warnings (more than the trunk's current 447 warnings).

            All new files has proper header.

            Unit test pass.

            Committed to trunk, 0.10 branch and 0.9 branch.

            daijy Daniel Dai added a comment - PIG-2125 -10.patch, PIG-2125 -10_0.9.patch fix test-patch warnings: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 46 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 462 release audit warnings (more than the trunk's current 447 warnings). All new files has proper header. Unit test pass. Committed to trunk, 0.10 branch and 0.9 branch.
            daijy Daniel Dai added a comment -

            After PIG-2125-10.patch, we can compile and run end-to-end tests on 23. There are still ~20 unit tests fail on 23 and we are working on that.

            Since all patches attached to the Jira is committed, and this thread is becoming big and confusing, I would suggest to close this ticket and create a separate ticket to track unit test fixes for hadoop 23, agree?

            daijy Daniel Dai added a comment - After PIG-2125 -10.patch, we can compile and run end-to-end tests on 23. There are still ~20 unit tests fail on 23 and we are working on that. Since all patches attached to the Jira is committed, and this thread is becoming big and confusing, I would suggest to close this ticket and create a separate ticket to track unit test fixes for hadoop 23, agree?
            daijy Daniel Dai added a comment -

            Move the remaining work to PIG-2347.

            daijy Daniel Dai added a comment - Move the remaining work to PIG-2347 .
            nkollar Nándor Kollár added a comment - - edited

            It looks like in this change two services are registered in META-INF/services, which are implemented in Hadoop client dependency. daijy do you remember why are these services registered in Pig an not just in Hadoop mapreduce client? It looks like due to this service registration and MAPREDUCE-6473 Pig related unit test break in Hive.

            nkollar Nándor Kollár added a comment - - edited It looks like in this change two services are registered in META-INF/services, which are implemented in Hadoop client dependency. daijy do you remember why are these services registered in Pig an not just in Hadoop mapreduce client? It looks like due to this service registration and MAPREDUCE-6473 Pig related unit test break in Hive.
            daijy Daniel Dai added a comment -

            org.apache.pig.ExecType is required, Pig use this to enumerate all exectype supported. org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider is used to work around an issue in Hadoop23, I think probably this is fixed in newer Hadoop and we can remove this if tested working.

            daijy Daniel Dai added a comment - org.apache.pig.ExecType is required, Pig use this to enumerate all exectype supported. org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider is used to work around an issue in Hadoop23, I think probably this is fixed in newer Hadoop and we can remove this if tested working.

            How can I make sure that if I remove the ClientProtocolProvider, it won't break the Hadoop integration? Is there a specific unit test for that, or I should execute the entire test suite (which is takes a lot of time)?

            nkollar Nándor Kollár added a comment - How can I make sure that if I remove the ClientProtocolProvider, it won't break the Hadoop integration? Is there a specific unit test for that, or I should execute the entire test suite (which is takes a lot of time)?
            daijy Daniel Dai added a comment -

            If something went wrong, I believe any MiniCluster or local unit test will fail. But for this change, we definitely need to run through all unit tests and system tests cuz the consequence is not obvious.

            daijy Daniel Dai added a comment - If something went wrong, I believe any MiniCluster or local unit test will fail. But for this change, we definitely need to run through all unit tests and system tests cuz the consequence is not obvious.

            People

              daijy Daniel Dai
              daijy Daniel Dai
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: