Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.4.0
    • None
    • HDDS BadLands

    Description

      In a secure Ozone cluster. OzoneManager fail to connect to SCM on SCMBlockLocationProtocol.

      Attachments

        1. HDDS-1039.00.patch
          0.8 kB
          Ajay Kumar

        Issue Links

          Activity

            ajayydv Ajay Kumar added a comment -
            2019-01-31 20:17:04,246 DEBUG ipc.Server (Server.java:saslProcess(1841)) - SASL server context established. Negotiated QoP is auth
            2019-01-31 20:17:04,247 DEBUG ipc.Server (Server.java:saslProcess(1846)) - SASL server successfully authenticated client: om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM (auth:KERBEROS)
            2019-01-31 20:17:04,247 DEBUG ipc.Server (Server.java:processResponse(1468)) - Socket Reader #1 for port 9863: responding to Call#-33 Retry#-1 null from 172.27.27.138:39305
            2019-01-31 20:17:04,247 DEBUG ipc.Server (Server.java:processResponse(1487)) - Socket Reader #1 for port 9863: responding to Call#-33 Retry#-1 null from 172.27.27.138:39305 Wrote 22 bytes.
            2019-01-31 20:17:04,255 DEBUG ipc.Server (Server.java:processOneRpc(2355)) -  got #-3
            2019-01-31 20:17:04,256 INFO  ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 172.27.27.138:39305 for protocol org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol is unauthorized for user om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM (auth:KERBEROS)
            2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:processOneRpc(2372)) - Socket Reader #1 for port 9863: processOneRpc from client 172.27.27.138:39305 threw exception [org.apache.hadoop.security.authorize.AuthorizationException: Unknown protocol: org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol]
            2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:processResponse(1468)) - Socket Reader #1 for port 9863: responding to Call#-3 Retry#-1 null from 172.27.27.138:39305
            2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:processResponse(1487)) - Socket Reader #1 for port 9863: responding to Call#-3 Retry#-1 null from 172.27.27.138:39305 Wrote 160 bytes.
            2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:close(3438)) - Socket Reader #1 for port 9863: disconnecting client 172.27.27.138:39305. Number of active connections: 0
            2019-01-31 20:17:04,821 DEBUG ipc.Server (Server.java:processOneRpc(2355)) -  got #35
            2019-01-31 20:17:04,821 DEBUG ipc.Server (Server.java:run(2670)) - IPC Server handler 3 on 9861: Call#35 Retry#0 org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol.sendHeartbeat from 172.27.27.138:41805 for RpcKind RPC_PROTOCOL_BUFFER
            ajayydv Ajay Kumar added a comment - 2019-01-31 20:17:04,246 DEBUG ipc.Server (Server.java:saslProcess(1841)) - SASL server context established. Negotiated QoP is auth 2019-01-31 20:17:04,247 DEBUG ipc.Server (Server.java:saslProcess(1846)) - SASL server successfully authenticated client: om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM (auth:KERBEROS) 2019-01-31 20:17:04,247 DEBUG ipc.Server (Server.java:processResponse(1468)) - Socket Reader #1 for port 9863: responding to Call#-33 Retry#-1 null from 172.27.27.138:39305 2019-01-31 20:17:04,247 DEBUG ipc.Server (Server.java:processResponse(1487)) - Socket Reader #1 for port 9863: responding to Call#-33 Retry#-1 null from 172.27.27.138:39305 Wrote 22 bytes. 2019-01-31 20:17:04,255 DEBUG ipc.Server (Server.java:processOneRpc(2355)) - got #-3 2019-01-31 20:17:04,256 INFO ipc.Server (Server.java:authorizeConnection(2562)) - Connection from 172.27.27.138:39305 for protocol org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol is unauthorized for user om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM (auth:KERBEROS) 2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:processOneRpc(2372)) - Socket Reader #1 for port 9863: processOneRpc from client 172.27.27.138:39305 threw exception [org.apache.hadoop.security.authorize.AuthorizationException: Unknown protocol: org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol] 2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:processResponse(1468)) - Socket Reader #1 for port 9863: responding to Call#-3 Retry#-1 null from 172.27.27.138:39305 2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:processResponse(1487)) - Socket Reader #1 for port 9863: responding to Call#-3 Retry#-1 null from 172.27.27.138:39305 Wrote 160 bytes. 2019-01-31 20:17:04,256 DEBUG ipc.Server (Server.java:close(3438)) - Socket Reader #1 for port 9863: disconnecting client 172.27.27.138:39305. Number of active connections: 0 2019-01-31 20:17:04,821 DEBUG ipc.Server (Server.java:processOneRpc(2355)) - got #35 2019-01-31 20:17:04,821 DEBUG ipc.Server (Server.java:run(2670)) - IPC Server handler 3 on 9861: Call#35 Retry#0 org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol.sendHeartbeat from 172.27.27.138:41805 for RpcKind RPC_PROTOCOL_BUFFER
            ajayydv Ajay Kumar added a comment -

            OM startup logs

            org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unknown protocol: org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol
            	at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1198)
            	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1063)
            19/01/31 20:17:04 DEBUG ipc.Client: IPC Client (1231006815) connection to ctr-e139-1542663976389-57536-01-000002.hwx.site/172.27.11.75:9863 from om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM: closed
            19/01/31 20:17:04 DEBUG ipc.Client: IPC Client (1231006815) connection to ctr-e139-1542663976389-57536-01-000002.hwx.site/172.27.11.75:9863 from om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM: stopped, remaining connections 0
            19/01/31 20:17:04 ERROR om.OzoneManager: Failed to start the OzoneManager.
            org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unknown protocol: org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol
            	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
            	at org.apache.hadoop.ipc.Client.call(Client.java:1443)
            	at org.apache.hadoop.ipc.Client.call(Client.java:1353)
            	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
            	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
            	at com.sun.proxy.$Proxy14.getScmInfo(Unknown Source)
            	at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.getScmInfo(ScmBlockLocationProtocolClientSideTranslatorPB.java:154)
            	at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:242)
            	at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:679)
            	at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:592)
            ajayydv Ajay Kumar added a comment - OM startup logs org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unknown protocol: org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1198) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1063) 19/01/31 20:17:04 DEBUG ipc.Client: IPC Client (1231006815) connection to ctr-e139-1542663976389-57536-01-000002.hwx.site/172.27.11.75:9863 from om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM: closed 19/01/31 20:17:04 DEBUG ipc.Client: IPC Client (1231006815) connection to ctr-e139-1542663976389-57536-01-000002.hwx.site/172.27.11.75:9863 from om/ctr-e139-1542663976389-57536-01-000003.hwx.site@EXAMPLE.COM: stopped, remaining connections 0 19/01/31 20:17:04 ERROR om.OzoneManager: Failed to start the OzoneManager. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unknown protocol: org.apache.hadoop.ozone.protocol.ScmBlockLocationProtocol at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy14.getScmInfo(Unknown Source) at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.getScmInfo(ScmBlockLocationProtocolClientSideTranslatorPB.java:154) at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:242) at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:679) at org.apache.hadoop.ozone.om.OzoneManager.main(OzoneManager.java:592)
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 16s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                  trunk Compile Tests
            +1 mvninstall 2m 37s trunk passed
            +1 checkstyle 0m 9s trunk passed
            0 findbugs 0m 0s Skipped patched modules with no Java source: .
            +1 findbugs 0m 0s trunk passed
            +1 javadoc 1m 31s trunk passed
                  Patch Compile Tests
            +1 mvninstall 2m 34s the patch passed
            +1 checkstyle 0m 9s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            0 findbugs 0m 0s Skipped patched modules with no Java source: .
            +1 findbugs 0m 0s the patch passed
            +1 javadoc 1m 25s the patch passed
                  Other Tests
            +1 unit 36m 5s hadoop-ozone in the patch passed.
            +1 unit 5m 33s hadoop-hdds in the patch passed.
            +1 asflicense 0m 19s The patch does not generate ASF License warnings.
            51m 3s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f
            JIRA Issue HDDS-1039
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12957170/HDDS-1039.00.patch
            Optional Tests asflicense javac javadoc unit findbugs checkstyle
            uname Linux a286ea43747d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh
            git revision trunk / f738b39
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_191
            Test Results https://builds.apache.org/job/PreCommit-HDDS-Build/2160/testReport/
            Max. process+thread count 1134 (vs. ulimit of 10000)
            modules C: hadoop-hdds/common U: hadoop-hdds/common
            Console output https://builds.apache.org/job/PreCommit-HDDS-Build/2160/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 2m 37s trunk passed +1 checkstyle 0m 9s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 0m 0s trunk passed +1 javadoc 1m 31s trunk passed       Patch Compile Tests +1 mvninstall 2m 34s the patch passed +1 checkstyle 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 0m 0s the patch passed +1 javadoc 1m 25s the patch passed       Other Tests +1 unit 36m 5s hadoop-ozone in the patch passed. +1 unit 5m 33s hadoop-hdds in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 51m 3s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f JIRA Issue HDDS-1039 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12957170/HDDS-1039.00.patch Optional Tests asflicense javac javadoc unit findbugs checkstyle uname Linux a286ea43747d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh git revision trunk / f738b39 maven version: Apache Maven 3.3.9 Default Java 1.8.0_191 Test Results https://builds.apache.org/job/PreCommit-HDDS-Build/2160/testReport/ Max. process+thread count 1134 (vs. ulimit of 10000) modules C: hadoop-hdds/common U: hadoop-hdds/common Console output https://builds.apache.org/job/PreCommit-HDDS-Build/2160/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            arp Arpit Agarwal added a comment -

            +1. Thanks for fixing this ajayydv.

            arp Arpit Agarwal added a comment - +1. Thanks for fixing this ajayydv .
            ajayydv Ajay Kumar added a comment -

            arpitagarwal thanks for review. Committed to trunk.

            ajayydv Ajay Kumar added a comment - arpitagarwal thanks for review. Committed to trunk.
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15881 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15881/)
            HDDS-1039. OzoneManager fails to connect with secure SCM. Contributed by (ajay: rev 9a19d6d48b463d4e2a598f0791d05fd2006a597c)

            • (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolPB.java
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15881 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15881/ ) HDDS-1039 . OzoneManager fails to connect with secure SCM. Contributed by (ajay: rev 9a19d6d48b463d4e2a598f0791d05fd2006a597c) (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolPB.java

            People

              ajayydv Ajay Kumar
              ajayydv Ajay Kumar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: