Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.1.2
-
None
Description
Came across this in our cluster:
1. The meta was assigned to a server 10.0.0.149,16020,1443507203340
2015-09-29 06:16:22,472 DEBUG [AM.ZK.Worker-pool2-t56] master.RegionStates: Onlined 1588230740 on 10.0.0.149,16020,1443507203340 {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2. The server dies at some point:
2015-09-29 06:18:25,952 INFO [main-EventThread] zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [10.0.0.149,16020,1443507203340] 2015-09-29 06:18:25,955 DEBUG [main-EventThread] master.AssignmentManager: based on AM, current region=hbase:meta,,1.1588230740 is on server=10.0.0.149,16020,1443507203340 server being checked: 10.0.0.149,16020,1443507203340
3. The balancer had computed a plan that contained a move for the meta:
2015-09-29 06:18:26,833 INFO [B.defaultRpcServer.handler=12,queue=0,port=16000] master.HMaster: balance hri=hbase:meta,,1.1588230740, src=10.0.0.149,16020,1443507203340, dest=10.0.0.205,16020,1443507257905
4. The following ensues after this, leading to the meta remaining unassigned:
2015-09-29 06:18:26,859 DEBUG [B.defaultRpcServer.handler=12,queue=0,port=16000] master.AssignmentManager: Offline hbase:meta,,1.1588230740, no need to unassign since it's on a dead server: 10.0.0.149,16020,1443507203340 ...................... 2015-09-29 06:18:26,899 INFO [B.defaultRpcServer.handler=12,queue=0,port=16000] master.RegionStates: Offlined 1588230740 from 10.0.0.149,16020,1443507203340 ..................... 2015-09-29 06:18:26,914 INFO [B.defaultRpcServer.handler=12,queue=0,port=16000] master.AssignmentManager: Skip assigning hbase:meta,,1.1588230740, it is on a dead but not processed yet server: 10.0.0.149,16020,1443507203340 .................... 2015-09-29 06:18:26,915 DEBUG [AM.ZK.Worker-pool2-t58] master.AssignmentManager: Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OFFLINE, ts=1443507506914, server=10.0.0.149,16020,1443507203340} .................... 2015-09-29 06:18:29,447 DEBUG [MASTER_META_SERVER_OPERATIONS-10.0.0.148:16000-2] master.AssignmentManager: based on AM, current region=hbase:meta,,1.1588230740 is on server=null server being checked: 10.0.0.149,16020,1443507203340 2015-09-29 06:18:29,451 INFO [MASTER_META_SERVER_OPERATIONS- 10.0.0.148:16000-2] handler.MetaServerShutdownHandler: META has been assigned to otherwhere, skip assigning. 2015-09-29 06:18:29,452 DEBUG [MASTER_META_SERVER_OPERATIONS-10.0.0.148:16000-2] master.DeadServer: Finished processing 10.0.0.149,16020,1443507203340
Attachments
Attachments
Issue Links
- relates to
-
HBASE-13330 Region left unassigned due to AM & SSH each thinking the assignment would be done by the other
- Closed