Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0-beta-2, 2.0.0
    • test
    • None

    Description

      The last two nightlies failed on this test in here on shutdown:

      2018-02-16 20:49:35,132 DEBUG [M:0;881c50037eea:35808] master.MasterRpcServices(1153): Checking to see if procedure is done pid=7
      2018-02-16 20:49:35,133 DEBUG [M:0;881c50037eea:35808] client.RpcRetryingCallerImpl(132): Call exception, tries=7, retries=7, started=8122 ms ago, cancelled=false, msg=null, details=, exception=org.apache.hadoop.hbase.MasterNotRunningException
        at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2736)
        at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:1155)
        at org.apache.hadoop.hbase.client.ShortCircuitMasterConnection.getProcedureResult(ShortCircuitMasterConnection.java:423)
        at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture$2.rpcCall(HBaseAdmin.java:3490)
        at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture$2.rpcCall(HBaseAdmin.java:3487)
        at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:100)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3055)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3047)
        at org.apache.hadoop.hbase.client.HBaseAdmin.access$700(HBaseAdmin.java:224)
        at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:3486)
        at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:3438)
        at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3394)
        at org.apache.hadoop.hbase.client.HBaseAdmin.get(HBaseAdmin.java:2123)
        at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:612)
        at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:586)
        at org.apache.hadoop.hbase.security.access.AccessController.createACLTable(AccessController.java:1130)
        at org.apache.hadoop.hbase.security.access.AccessController.postStartMaster(AccessController.java:1107)
        at org.apache.hadoop.hbase.master.MasterCoprocessorHost$71.call(MasterCoprocessorHost.java:994)
        at org.apache.hadoop.hbase.master.MasterCoprocessorHost$71.call(MasterCoprocessorHost.java:991)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
        at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:991)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:555)
        at java.lang.Thread.run(Thread.java:748)
      

      We get stuck retrying.

      Attachments

        1. HBASE-20013.branch-2.001.patch
          1 kB
          Michael Stack

        Issue Links

          Activity

            stack Michael Stack added a comment -

            .001 Throw a DoNotRetryException if Master is stopped instead of a MasterNotRunningException...... The latter means retries.

            Let me see how this does. Hard to make a test for a behavior so deep.

            stack Michael Stack added a comment - .001 Throw a DoNotRetryException if Master is stopped instead of a MasterNotRunningException...... The latter means retries. Let me see how this does. Hard to make a test for a behavior so deep.
            stack Michael Stack added a comment -

            Let me try hadoopqa first....

            stack Michael Stack added a comment - Let me try hadoopqa first....
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 15s Docker mode activated.
                  Prechecks
            0 findbugs 0m 0s Findbugs executables are not available.
            +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                  branch-2 Compile Tests
            +1 mvninstall 3m 17s branch-2 passed
            +1 compile 0m 41s branch-2 passed
            +1 checkstyle 1m 6s branch-2 passed
            +1 shadedjars 5m 3s branch has no errors when building our shaded downstream artifacts.
            +1 javadoc 0m 27s branch-2 passed
                  Patch Compile Tests
            +1 mvninstall 3m 22s the patch passed
            +1 compile 0m 42s the patch passed
            +1 javac 0m 42s the patch passed
            -1 checkstyle 1m 6s hbase-server: The patch generated 1 new + 173 unchanged - 1 fixed = 174 total (was 174)
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedjars 3m 58s patch has no errors when building our shaded downstream artifacts.
            +1 hadoopcheck 14m 38s Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0.
            +1 javadoc 0m 28s the patch passed
                  Other Tests
            +1 unit 103m 59s hbase-server in the patch passed.
            +1 asflicense 0m 20s The patch does not generate ASF License warnings.
            134m 31s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db
            JIRA Issue HBASE-20013
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12910994/HBASE-20013.branch-2.001.patch
            Optional Tests asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
            uname Linux f993c698a8c6 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux
            Build tool maven
            Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
            git revision branch-2 / 34d5f2b70e
            maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
            Default Java 1.8.0_151
            checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/11556/artifact/patchprocess/diff-checkstyle-hbase-server.txt
            Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11556/testReport/
            Max. process+thread count 5126 (vs. ulimit of 10000)
            modules C: hbase-server U: hbase-server
            Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11556/console
            Powered by Apache Yetus 0.7.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks 0 findbugs 0m 0s Findbugs executables are not available. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       branch-2 Compile Tests +1 mvninstall 3m 17s branch-2 passed +1 compile 0m 41s branch-2 passed +1 checkstyle 1m 6s branch-2 passed +1 shadedjars 5m 3s branch has no errors when building our shaded downstream artifacts. +1 javadoc 0m 27s branch-2 passed       Patch Compile Tests +1 mvninstall 3m 22s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -1 checkstyle 1m 6s hbase-server: The patch generated 1 new + 173 unchanged - 1 fixed = 174 total (was 174) +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedjars 3m 58s patch has no errors when building our shaded downstream artifacts. +1 hadoopcheck 14m 38s Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. +1 javadoc 0m 28s the patch passed       Other Tests +1 unit 103m 59s hbase-server in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 134m 31s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db JIRA Issue HBASE-20013 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12910994/HBASE-20013.branch-2.001.patch Optional Tests asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile uname Linux f993c698a8c6 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision branch-2 / 34d5f2b70e maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) Default Java 1.8.0_151 checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/11556/artifact/patchprocess/diff-checkstyle-hbase-server.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11556/testReport/ Max. process+thread count 5126 (vs. ulimit of 10000) modules C: hbase-server U: hbase-server Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11556/console Powered by Apache Yetus 0.7.0 http://yetus.apache.org This message was automatically generated.
            stack Michael Stack added a comment -

            Pushed to master and branch-2 after fixing the checkstyle warning.

            stack Michael Stack added a comment - Pushed to master and branch-2 after fixing the checkstyle warning.
            stack Michael Stack added a comment -

            Leaving open in case I've not fixed the issue.

            stack Michael Stack added a comment - Leaving open in case I've not fixed the issue.
            hudson Hudson added a comment -

            FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4599 (See https://builds.apache.org/job/HBase-Trunk_matrix/4599/)
            HBASE-20013 TestZKPermissionWatcher is flakey (stack: rev 68d509bc1ff7a3bf69a596aed49f238b42ee0679)

            • (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
            hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4599 (See https://builds.apache.org/job/HBase-Trunk_matrix/4599/ ) HBASE-20013 TestZKPermissionWatcher is flakey (stack: rev 68d509bc1ff7a3bf69a596aed49f238b42ee0679) (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
            stack Michael Stack added a comment -

            Resolving. No longer on flakies list (HBASE-20020 will probably help here too...)

            stack Michael Stack added a comment - Resolving. No longer on flakies list ( HBASE-20020 will probably help here too...)
            chia7712 Chia-Ping Tsai added a comment -

            It is still flaky in branch-1 and branch-1.4. Should we reopen this? Or file another Jira to trace it for branch-1?

            chia7712 Chia-Ping Tsai added a comment - It is still flaky in branch-1 and branch-1.4. Should we reopen this? Or file another Jira to trace it for branch-1?
            stack Michael Stack added a comment -

            chia7712 I opened HBASE-20025 for backport (Trying to keep the branch-2 JIRA number count down as I delude myself into thinking hbase2 is almost done – smile). Thanks sir.

            stack Michael Stack added a comment - chia7712 I opened HBASE-20025 for backport (Trying to keep the branch-2 JIRA number count down as I delude myself into thinking hbase2 is almost done – smile). Thanks sir.

            People

              stack Michael Stack
              stack Michael Stack
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: