Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19147 All branch-2 unit tests pass
  3. HBASE-19220

Async tests time out talking to zk; 'clusterid came back null'

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0-beta-1, 2.0.0
    • test
    • None
    • Changed retries from 3 to 30 for zk initial connect for registry.

    Description

      I see this in test runs on a dedicated machine:

      [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 652.514 s <<< FAILURE! - in org.apache.hadoop.hbase.client.TestAsyncAdminBuilder
      [ERROR] testRpcTimeout[0](org.apache.hadoop.hbase.client.TestAsyncAdminBuilder) Time elapsed: 213.618 s <<< ERROR!
      java.util.concurrent.ExecutionException: java.io.IOException: clusterid came back null
      at org.apache.hadoop.hbase.client.TestAsyncAdminBuilder.testRpcTimeout(TestAsyncAdminBuilder.java:105)
      Caused by: java.io.IOException: clusterid came back null

      [ERROR] org.apache.hadoop.hbase.client.TestAsyncTableScanMetrics Time elapsed: 0.007 s <<< ERROR!
      java.util.concurrent.ExecutionException: java.io.IOException: clusterid came back null
      at org.apache.hadoop.hbase.client.TestAsyncTableScanMetrics.setUp(TestAsyncTableScanMetrics.java:97)
      Caused by: java.io.IOException: clusterid came back null

      [ERROR] org.apache.hadoop.hbase.client.TestRawAsyncScanCursor Time elapsed: 0.005 s <<< ERROR!
      java.util.concurrent.ExecutionException: java.io.IOException: clusterid came back null
      at org.apache.hadoop.hbase.client.TestRawAsyncScanCursor.setUpBeforeClass(TestRawAsyncScanCursor.java:42)
      Caused by: java.io.IOException: clusterid came back null

      [ERROR] org.apache.hadoop.hbase.client.TestAsyncNamespaceAdminApi Time elapsed: 0.005 s <<< ERROR!
      java.util.concurrent.ExecutionException: java.io.IOException: clusterid came back null
      at org.apache.hadoop.hbase.client.TestAsyncNamespaceAdminApi.setUpBeforeClass(TestAsyncNamespaceAdminApi.java:66)
      Caused by: java.io.IOException: clusterid came back null

      If I up the retries, they go away.

      At least on this machine, I notice that zk connections can take a while... see HBASE-19102 where we add a wait on the Connection to come up before progressing.

      Suggest that I up the retries. No harm in trying more. It is currently set to 3 retries at a one second interval.

      Attachments

        1. 19220.patch
          1.0 kB
          Michael Stack

        Activity

          stack Michael Stack added a comment -

          FYI Apache9 Ok I commit this?

          stack Michael Stack added a comment - FYI Apache9 Ok I commit this?
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 7s Docker mode activated.
                Prechecks
          0 findbugs 0m 0s Findbugs executables are not available.
          +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                master Compile Tests
          +1 mvninstall 4m 44s master passed
          +1 compile 0m 20s master passed
          +1 checkstyle 0m 26s master passed
          +1 shadedjars 5m 5s branch has no errors when building our shaded downstream artifacts.
          +1 javadoc 0m 19s master passed
                Patch Compile Tests
          +1 mvninstall 4m 26s the patch passed
          +1 compile 0m 19s the patch passed
          +1 javac 0m 19s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedjars 4m 37s patch has no errors when building our shaded downstream artifacts.
          +1 hadoopcheck 49m 35s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4.
          +1 javadoc 0m 19s the patch passed
                Other Tests
          +1 unit 2m 37s hbase-client in the patch passed.
          +1 asflicense 0m 9s The patch does not generate ASF License warnings.
          68m 39s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01
          JIRA Issue HBASE-19220
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896804/19220.patch
          Optional Tests asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
          uname Linux 33940b80ef46 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
          git revision master / cffbe4bef7
          maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
          Default Java 1.8.0_151
          Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/9704/testReport/
          modules C: hbase-client U: hbase-client
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/9704/console
          Powered by Apache Yetus 0.6.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s Docker mode activated.       Prechecks 0 findbugs 0m 0s Findbugs executables are not available. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       master Compile Tests +1 mvninstall 4m 44s master passed +1 compile 0m 20s master passed +1 checkstyle 0m 26s master passed +1 shadedjars 5m 5s branch has no errors when building our shaded downstream artifacts. +1 javadoc 0m 19s master passed       Patch Compile Tests +1 mvninstall 4m 26s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed +1 checkstyle 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedjars 4m 37s patch has no errors when building our shaded downstream artifacts. +1 hadoopcheck 49m 35s Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. +1 javadoc 0m 19s the patch passed       Other Tests +1 unit 2m 37s hbase-client in the patch passed. +1 asflicense 0m 9s The patch does not generate ASF License warnings. 68m 39s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 JIRA Issue HBASE-19220 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896804/19220.patch Optional Tests asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile uname Linux 33940b80ef46 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / cffbe4bef7 maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) Default Java 1.8.0_151 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/9704/testReport/ modules C: hbase-client U: hbase-client Console output https://builds.apache.org/job/PreCommit-HBASE-Build/9704/console Powered by Apache Yetus 0.6.0 http://yetus.apache.org This message was automatically generated.
          zhangduo Duo Zhang added a comment -

          Is it a good idea to change a config in normal code path which only aims to make a UT pass?

          Anyway, I think we can commit this first to make our UTs more stable. And open another issue to tune the retry configs.

          Thanks.

          zhangduo Duo Zhang added a comment - Is it a good idea to change a config in normal code path which only aims to make a UT pass? Anyway, I think we can commit this first to make our UTs more stable. And open another issue to tune the retry configs. Thanks.
          stack Michael Stack added a comment -

          Push to branch-2 and master.

          stack Michael Stack added a comment - Push to branch-2 and master.
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4027 (See https://builds.apache.org/job/HBase-Trunk_matrix/4027/)
          HBASE-19220 Async tests time out talking to zk; 'clusterid came back (stack: rev 252ab30820c74409186823a672672d8a03b940ca)

          • (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4027 (See https://builds.apache.org/job/HBase-Trunk_matrix/4027/ ) HBASE-19220 Async tests time out talking to zk; 'clusterid came back (stack: rev 252ab30820c74409186823a672672d8a03b940ca) (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-2.0 #828 (See https://builds.apache.org/job/HBase-2.0/828/)
          HBASE-19220 Async tests time out talking to zk; 'clusterid came back (stack: rev afdafb4bad6de47c75c57edfc2e53c9f7d487abd)

          • (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-2.0 #828 (See https://builds.apache.org/job/HBase-2.0/828/ ) HBASE-19220 Async tests time out talking to zk; 'clusterid came back (stack: rev afdafb4bad6de47c75c57edfc2e53c9f7d487abd) (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java

          People

            stack Michael Stack
            stack Michael Stack
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: