Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
1.3.1
-
None
-
None
Description
Snapshot creation failed with below error when tried on table with multiple replica region,
hbase(main):025:0> snapshot 't1','t1_snap' 2017-11-16 18:04:27,930 DEBUG [main] client.HBaseAdmin: Waiting a max of 300000 ms for snapshot '{ ss=t1_snap table=t1 type=FLUSH }'' to complete. (max 42857 ms per retry) 2017-11-16 18:04:27,930 DEBUG [main] client.HBaseAdmin: (#1) Sleeping: 100ms while waiting for snapshot completion. 2017-11-16 18:04:28,030 DEBUG [main] client.HBaseAdmin: Getting current status of snapshot from master... 2017-11-16 18:04:28,035 DEBUG [main] client.HBaseAdmin: (#2) Sleeping: 200ms while waiting for snapshot completion. 2017-11-16 18:04:28,236 DEBUG [main] client.HBaseAdmin: Getting current status of snapshot from master... 2017-11-16 18:04:28,238 DEBUG [main] client.HBaseAdmin: (#3) Sleeping: 300ms while waiting for snapshot completion. 2017-11-16 18:04:28,538 DEBUG [main] client.HBaseAdmin: Getting current status of snapshot from master... ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=t1_snap table=t1 type=FLUSH } had an error. Procedure t1_snap { waiting=[] done=[] } at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:354) at org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1091) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2418) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:191) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException via Failed taking snapshot { ss=t1_snap table=t1 type=FLUSH } due to exception:Manifest region info {ENCODED => 3158abebd655fca73cd87b6e84584197, NAME => 't1,,1510826577196_0002.3158abebd655fca73cd87b6e84584197.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true, REPLICA_ID => 2}doesn't match expected region:{ENCODED => 73aa1a133d3344a67afa46ee135e389a, NAME => 't1,,1510826577196.73aa1a133d3344a67afa46ee135e389a.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true}:org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Manifest region info {ENCODED => 3158abebd655fca73cd87b6e84584197, NAME => 't1,,1510826577196_0002.3158abebd655fca73cd87b6e84584197.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true, REPLICA_ID => 2}doesn't match expected region:{ENCODED => 73aa1a133d3344a67afa46ee135e389a, NAME => 't1,,1510826577196.73aa1a133d3344a67afa46ee135e389a.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true} at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83) at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:315) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:344) ... 6 more Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Manifest region info {ENCODED => 3158abebd655fca73cd87b6e84584197, NAME => 't1,,1510826577196_0002.3158abebd655fca73cd87b6e84584197.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true, REPLICA_ID => 2}doesn't match expected region:{ENCODED => 73aa1a133d3344a67afa46ee135e389a, NAME => 't1,,1510826577196.73aa1a133d3344a67afa46ee135e389a.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true} at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegionInfo(MasterSnapshotVerifier.java:220) at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:198) at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:118) at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:202) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Steps to reproduce:
1. create 't1','cf1', {REGION_REPLICATION => 3} 2. put 't1','r1','cf1:C1','1' 3. split 't1','m' 4. snapshot 't1','t1_snap'
Attachments
Attachments
Issue Links
- Is contained by
-
HBASE-18223 Track the effort to improve/bug fix read replica feature
- Resolved