Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23175

Yarn unable to acquire delegation token for HBase Spark jobs

Details

    • Reviewed

    Description

      Spark rely on the TokenUtil.obtainToken(conf) API which is removed in HBase-2.0, though it has been fixed in SPARK-26432 to use the new API but planned for Spark-3.0, hence we need the fix in HBase until they release it and we upgrade it

      18/03/20 20:39:07 ERROR ApplicationMaster: User class threw exception: org.apache.hadoop.hbase.HBaseIOException: com.google.protobuf.ServiceException: Error calling method hbase.pb.AuthenticationService.GetAuthenticationToken
      org.apache.hadoop.hbase.HBaseIOException: com.google.protobuf.ServiceException: Error calling method hbase.pb.AuthenticationService.GetAuthenticationToken
              at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
              at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:346)
              at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
              at org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:121)
              at org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:118)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
              at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
              at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:118)
              at org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:272)
              at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:533)
              at org.apache.hadoop.hbase.spark.HBaseContext.<init>(HBaseContext.scala:73)
              at org.apache.hadoop.hbase.spark.JavaHBaseContext.<init>(JavaHBaseContext.scala:46)
              at org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkDeleteExample.main(JavaHBaseBulkDeleteExample.java:64)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
      Caused by: com.google.protobuf.ServiceException: Error calling method hbase.pb.AuthenticationService.GetAuthenticationToken
              at org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:71)
              at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
              at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:81)
              ... 17 more
      

      Attachments

        1. HBASE-23175.master.001.patch
          1 kB
          Ankit Singhal

        Activity

          ankit@apache.org Ankit Singhal added a comment -

          Will raise an amendment ticket for SPARK-26432 separately as the new change is relying on the API which is deprecated recently.
          elserj, can you please review the patch.

          ankit@apache.org Ankit Singhal added a comment - Will raise an amendment ticket for SPARK-26432 separately as the new change is relying on the API which is deprecated recently. elserj , can you please review the patch.
          zhangduo Duo Zhang added a comment -

          So we will add this method back on all branch-2.x? Do we need this on master? Master is for 3.0.0.

          zhangduo Duo Zhang added a comment - So we will add this method back on all branch-2.x? Do we need this on master? Master is for 3.0.0.
          ankit@apache.org Ankit Singhal added a comment -

          So we will add this method back on all branch-2.x? Do we need this on master? Master is for 3.0.0.

          yes, we need this in all branch-2.x and master.

          ankit@apache.org Ankit Singhal added a comment - So we will add this method back on all branch-2.x? Do we need this on master? Master is for 3.0.0. yes, we need this in all branch-2.x and master.
          elserj Josh Elser added a comment -

          Will raise an amendment ticket for SPARK-26432 separately as the new change is relying on the API which is deprecated recently.

          The change looks straightforward enough and reasonable for us to make to improve our spark compatibility.

          Thinking forward to release notes, do you have an idea of what versions of Spark 2 would be incompatible with HBase 2.x? Is it all Spark2.x against all currently-release HBase 2.x?

          elserj Josh Elser added a comment - Will raise an amendment ticket for SPARK-26432 separately as the new change is relying on the API which is deprecated recently. The change looks straightforward enough and reasonable for us to make to improve our spark compatibility. Thinking forward to release notes, do you have an idea of what versions of Spark 2 would be incompatible with HBase 2.x? Is it all Spark2.x against all currently-release HBase 2.x?
          ankit@apache.org Ankit Singhal added a comment -

          do you have an idea of what versions of Spark 2 would be incompatible with HBase 2.x? Is it all Spark2.x against all currently-release HBase 2.x?

          yes all versions of HBase 2.x against all versions of Spark 2.x.

          ankit@apache.org Ankit Singhal added a comment - do you have an idea of what versions of Spark 2 would be incompatible with HBase 2.x? Is it all Spark2.x against all currently-release HBase 2.x? yes all versions of HBase 2.x against all versions of Spark 2.x.
          elserj Josh Elser added a comment -

          I just kicked QA by hand to run. Reintroducing the deprecated method is the easiest thing we can do to let the breadth of Spark 2 releases actually use HBase. Thanks for getting Spark onto the newer API, Ankit. Good on you.

          +1 when QA comes back.

          elserj Josh Elser added a comment - I just kicked QA by hand to run. Reintroducing the deprecated method is the easiest thing we can do to let the breadth of Spark 2 releases actually use HBase. Thanks for getting Spark onto the newer API, Ankit. Good on you. +1 when QA comes back.
          HBaseQA HBase QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 1m 10s Docker mode activated.
                Prechecks
          +1 dupname 0m 0s No case conflicting files found.
          +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -0 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                master Compile Tests
          +1 mvninstall 5m 38s master passed
          +1 compile 0m 56s master passed
          +1 checkstyle 1m 30s master passed
          +1 shadedjars 5m 1s branch has no errors when building our shaded downstream artifacts.
          +1 javadoc 0m 32s master passed
          0 spotbugs 4m 21s Used deprecated FindBugs config; considering switching to SpotBugs.
          +1 findbugs 4m 17s master passed
                Patch Compile Tests
          +1 mvninstall 5m 23s the patch passed
          +1 compile 0m 54s the patch passed
          +1 javac 0m 54s the patch passed
          -1 checkstyle 1m 28s hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedjars 5m 0s patch has no errors when building our shaded downstream artifacts.
          +1 hadoopcheck 17m 5s Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
          +1 javadoc 0m 33s the patch passed
          +1 findbugs 4m 24s the patch passed
                Other Tests
          -1 unit 229m 42s hbase-server in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          290m 5s



          Reason Tests
          Failed junit tests hadoop.hbase.master.TestMasterShutdown



          Subsystem Report/Notes
          Docker Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/PreCommit-HBASE-Build/973/artifact/patchprocess/Dockerfile
          JIRA Issue HBASE-23175
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12983008/HBASE-23175.master.001.patch
          Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
          uname Linux 1fcc6e351b56 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux
          Build tool maven
          Personality dev-support/hbase-personality.sh
          git revision master / 65ee17086a
          Default Java 1.8.0_181
          checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/973/artifact/patchprocess/diff-checkstyle-hbase-server.txt
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/973/artifact/patchprocess/patch-unit-hbase-server.txt
          Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/973/testReport/
          Max. process+thread count 4975 (vs. ulimit of 10000)
          modules C: hbase-server U: hbase-server
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/973/console
          versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
          Powered by Apache Yetus 0.11.0 https://yetus.apache.org

          This message was automatically generated.

          HBaseQA HBase QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 10s Docker mode activated.       Prechecks +1 dupname 0m 0s No case conflicting files found. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. -0 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       master Compile Tests +1 mvninstall 5m 38s master passed +1 compile 0m 56s master passed +1 checkstyle 1m 30s master passed +1 shadedjars 5m 1s branch has no errors when building our shaded downstream artifacts. +1 javadoc 0m 32s master passed 0 spotbugs 4m 21s Used deprecated FindBugs config; considering switching to SpotBugs. +1 findbugs 4m 17s master passed       Patch Compile Tests +1 mvninstall 5m 23s the patch passed +1 compile 0m 54s the patch passed +1 javac 0m 54s the patch passed -1 checkstyle 1m 28s hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedjars 5m 0s patch has no errors when building our shaded downstream artifacts. +1 hadoopcheck 17m 5s Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. +1 javadoc 0m 33s the patch passed +1 findbugs 4m 24s the patch passed       Other Tests -1 unit 229m 42s hbase-server in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 290m 5s Reason Tests Failed junit tests hadoop.hbase.master.TestMasterShutdown Subsystem Report/Notes Docker Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/PreCommit-HBASE-Build/973/artifact/patchprocess/Dockerfile JIRA Issue HBASE-23175 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12983008/HBASE-23175.master.001.patch Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile uname Linux 1fcc6e351b56 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux Build tool maven Personality dev-support/hbase-personality.sh git revision master / 65ee17086a Default Java 1.8.0_181 checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/973/artifact/patchprocess/diff-checkstyle-hbase-server.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/973/artifact/patchprocess/patch-unit-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/973/testReport/ Max. process+thread count 4975 (vs. ulimit of 10000) modules C: hbase-server U: hbase-server Console output https://builds.apache.org/job/PreCommit-HBASE-Build/973/console versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 Powered by Apache Yetus 0.11.0 https://yetus.apache.org This message was automatically generated.
          elserj Josh Elser added a comment -

          Fixed the checkstyle locally (added the deprecated javadoc annotation to the existing comment) and will push. Thanks Ankit.

          elserj Josh Elser added a comment - Fixed the checkstyle locally (added the deprecated javadoc annotation to the existing comment) and will push. Thanks Ankit.
          elserj Josh Elser added a comment -

          Actually, ankit@apache.org, looks like HBASE-23194 landed since QA ran which invalidates this patch.

          elserj Josh Elser added a comment - Actually, ankit@apache.org , looks like  HBASE-23194 landed since QA ran which invalidates this patch.
          ankit@apache.org Ankit Singhal added a comment -

          yeah, HBASE-23194 removes all the deprecated methods from InterfaceAudiende.Public class, don't we remove methods from Public classes until the 1-2 major releases?

          ankit@apache.org Ankit Singhal added a comment - yeah, HBASE-23194 removes all the deprecated methods from InterfaceAudiende.Public class, don't we remove methods from Public classes until the 1-2 major releases?
          ankit@apache.org Ankit Singhal added a comment -

          Thanks elserj for pursuing it, as HBASE-23194 reverted, can we have this committed now?

          ankit@apache.org Ankit Singhal added a comment - Thanks elserj for pursuing it, as HBASE-23194 reverted, can we have this committed now?
          stack Michael Stack added a comment -

          LGTM. I can commit if good by you elserj sir.

          stack Michael Stack added a comment - LGTM. I can commit if good by you elserj sir.
          elserj Josh Elser added a comment -

          Please do. Thanks Stack.

          elserj Josh Elser added a comment - Please do. Thanks Stack.
          stack Michael Stack added a comment -

          Pushed on branch-2.1+ after fixing checkstyle and checking test failure (it passed for me). Thanks for patch ankit@apache.org

          stack Michael Stack added a comment - Pushed on branch-2.1+ after fixing checkstyle and checking test failure (it passed for me). Thanks for patch ankit@apache.org
          ankit@apache.org Ankit Singhal added a comment -

          Thanks stack

          ankit@apache.org Ankit Singhal added a comment - Thanks stack
          hudson Hudson added a comment -

          Results for branch branch-2.1
          build #1695 on builds.a.o: +1 overall


          details (if available):

          +1 general checks
          – For more information see general report

          +1 jdk8 hadoop2 checks
          – For more information see jdk8 (hadoop2) report

          +1 jdk8 hadoop3 checks
          – For more information see jdk8 (hadoop3) report

          +1 source release artifact
          – See build output for details.

          +1 client integration test

          hudson Hudson added a comment - Results for branch branch-2.1 build #1695 on builds.a.o : +1 overall details (if available): +1 general checks – For more information see general report +1 jdk8 hadoop2 checks – For more information see jdk8 (hadoop2) report +1 jdk8 hadoop3 checks – For more information see jdk8 (hadoop3) report +1 source release artifact – See build output for details. +1 client integration test
          hudson Hudson added a comment -

          Results for branch master
          build #1522 on builds.a.o: -1 overall


          details (if available):

          +1 general checks
          – For more information see general report

          +1 jdk8 hadoop2 checks
          – For more information see jdk8 (hadoop2) report

          -1 jdk8 hadoop3 checks
          – Something went wrong running this stage, please check relevant console output.

          +1 source release artifact
          – See build output for details.

          +1 client integration test

          hudson Hudson added a comment - Results for branch master build #1522 on builds.a.o : -1 overall details (if available): +1 general checks – For more information see general report +1 jdk8 hadoop2 checks – For more information see jdk8 (hadoop2) report -1 jdk8 hadoop3 checks – Something went wrong running this stage, please check relevant console output . +1 source release artifact – See build output for details. +1 client integration test
          hudson Hudson added a comment -

          Results for branch branch-2
          build #2340 on builds.a.o: -1 overall


          details (if available):

          +1 general checks
          – For more information see general report

          -1 jdk8 hadoop2 checks
          – For more information see jdk8 (hadoop2) report

          -1 jdk8 hadoop3 checks
          – For more information see jdk8 (hadoop3) report

          +1 source release artifact
          – See build output for details.

          +1 client integration test

          hudson Hudson added a comment - Results for branch branch-2 build #2340 on builds.a.o : -1 overall details (if available): +1 general checks – For more information see general report -1 jdk8 hadoop2 checks – For more information see jdk8 (hadoop2) report -1 jdk8 hadoop3 checks – For more information see jdk8 (hadoop3) report +1 source release artifact – See build output for details. +1 client integration test
          zghao Guanghao Zhang added a comment -

          Forgot to pushed to branch-2.2. Reopened it.

          zghao Guanghao Zhang added a comment - Forgot to pushed to branch-2.2. Reopened it.
          zghao Guanghao Zhang added a comment -

          Pushed to branch-2.2.

          zghao Guanghao Zhang added a comment - Pushed to branch-2.2.
          hudson Hudson added a comment -

          Results for branch branch-2.2
          build #743 on builds.a.o: -1 overall


          details (if available):

          +1 general checks
          – For more information see general report

          -1 jdk8 hadoop2 checks
          – For more information see jdk8 (hadoop2) report

          -1 jdk8 hadoop3 checks
          – For more information see jdk8 (hadoop3) report

          +1 source release artifact
          – See build output for details.

          +1 client integration test

          hudson Hudson added a comment - Results for branch branch-2.2 build #743 on builds.a.o : -1 overall details (if available): +1 general checks – For more information see general report -1 jdk8 hadoop2 checks – For more information see jdk8 (hadoop2) report -1 jdk8 hadoop3 checks – For more information see jdk8 (hadoop3) report +1 source release artifact – See build output for details. +1 client integration test

          People

            ankit@apache.org Ankit Singhal
            ankit@apache.org Ankit Singhal
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: