Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-10614

Master could not be stopped

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.94.16, 0.99.0
    • 0.96.2, 0.98.1, 0.99.0, 0.94.18
    • master
    • None
    • Reviewed

    Description

      It's an issue when to run "bin/hbase master stop" to shutdown the cluster.
      This could be reproduced by the following steps. Particularly for the trunk code, we need to configure the hbase.assignment.maximum.attempts as 1.
      1. Start one master and several region servers.
      2. Stop all the region servers.
      3. After a while, run "bin/hbase master stop" to shutdown the cluster.

      As a result, the master could not be stopped within a short time, but will be stopped after several hours. And after it's stopped, i find the error logs.
      1. For the trunk:
      A. lots of the logs which are "java.io.IOException: Failed to find location, tableName=hbase:meta, row=, reload=true"
      B..And at last, there's one exception before the master is stopped, "ServerShutdownHandler: Received exception accessing hbase:meta during server shutdown of server-XXX, retrying hbase:meta read
      java.io.InterruptedIOException: Interrupted after 0 tries on 350."
      2. For the branch 0.94:
      A. lots of the logs which are "Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@44285d14; serverName=".
      B. And at last, there's one exception before the master is stopped, "ServerShutdownHandler: Received exception accessing META during server shutdown of server-XXX, retrying META read
      org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for after 140 tries."

      We could see the master are stopped after lots of reties which are not necessary when the cluster is shutdown.

      Attachments

        1. HBASE-10614.patch
          2 kB
          Jingcheng Du
        2. HBASE-10614-0.94.patch
          2 kB
          Jingcheng Du

        Activity

          People

            jingcheng.du@intel.com Jingcheng Du
            jingcheng.du@intel.com Jingcheng Du
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: