Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21621

Reversed scan does not return expected number of rows

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.0.0-alpha-1, 2.1.1
    • 3.0.0-alpha-1, 2.2.0, 2.1.2, 2.0.4
    • Scanners
    • None

    Description

      Steps to reproduce

      1. Create a table and put some data into it (data should be big enough, say N rows)
      2. Flush the table
      3. Scan the table with reversed set to true

      Expected Result
      N rows should be retrieved in reversed order

      Actual Result
      Less than expected number of rows is retrieved with following error in logs

      2018-12-19 21:55:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] regionserver.StoreScanner(1000): Switch to stream read (scanned=262214 bytes) of cf
      2018-12-19 21:55:32,955 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.RpcServer(471): Unexpected throwable object 
      java.lang.AssertionError: Key \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4 followed by a error order key \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in cf cf in reversed scan
      	at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.checkScanOrder(ReversedStoreScanner.java:105)
      	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:568)
      	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
      	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6598)
      	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6762)
      	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6535)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3252)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3501)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      2018-12-19 21:55:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142): callId: 508 service: ClientService methodName: Scan size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955, exception=java.io.IOException: Key \x00\x00\x00\x00\x00\x00\x00\x09/cf:a/1545236714675/Put/vlen=131072/seqid=4 followed by a error order key \x00\x00\x00\x00\x00\x00\x00\x0F/cf:a/1545236715545/Put/vlen=131072/seqid=8 in cf cf in reversed scan
      2018-12-19 21:55:33,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39007] ipc.CallRunner(142): callId: 511 service: ClientService methodName: Scan size: 47 connection: 127.0.0.1:48328 deadline: 1545236792955, exception=org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2421102592655360183 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false renew: false
      2018-12-19 21:55:33,060 DEBUG [Time-limited test] client.ScannerCallableWithReplicas(200): Scan with primary region returns org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 2421102592655360183 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false renew: false
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkScanNextCallSeq(RSRpcServices.java:3122)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3455)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      

      Analysis/Issue
      From initial analysis it seems problem occurs when we switch read type

      Attachments

        1. HBASE-21621.master.UT.patch
          3 kB
          Nihal Jain
        2. HBASE-21621.master.001.patch
          6 kB
          Guanghao Zhang
        3. HBASE-21621.master.002.patch
          7 kB
          Guanghao Zhang

        Issue Links

          Activity

            People

              zghao Guanghao Zhang
              nihaljain.cs Nihal Jain
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: