Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
3.0.0-alpha-1, 2.1.1
-
None
Description
Steps to reproduce
- Create a table and put some data into it (data should be big enough, say N rows)
- Flush the table
- Scan the table with reversed set to true
Expected Result
N rows should be retrieved in reversed order
Actual Result
Less than expected number of rows is retrieved with following error in logs
2018-12-19 21:55:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] regionserver.StoreScanner(1000): Switch to stream read (scanned=262214 bytes) of cf 2018-12-19 21:55:32,955 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.RpcServer(471): Unexpected throwable object java.lang.AssertionError: Key \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4 followed by a error order key \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in cf cf in reversed scan at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.checkScanOrder(ReversedStoreScanner.java:105) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:568) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6598) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6762) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6535) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3252) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3501) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-12-19 21:55:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142): callId: 508 service: ClientService methodName: Scan size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955, exception=java.io.IOException: Key \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4 followed by a error order key \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in cf cf in reversed scan 2018-12-19 21:55:33,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142): callId: 511 service: ClientService methodName: Scan size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955, exception=org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2421102592655360183 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false renew: false 2018-12-19 21:55:33,060 DEBUG [Time-limited test] client.ScannerCallableWithReplicas(200): Scan with primary region returns org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2421102592655360183 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false renew: false at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkScanNextCallSeq(RSRpcServices.java:3122) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3455) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Analysis/Issue
From initial analysis it seems problem occurs when we switch read type
Attachments
Attachments
Issue Links
- links to