Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21130

NullPointerException in StoreScanner

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 1.3.1
    • None
    • regionserver
    • None

    Description

      I've created a script that will put continuous put one record (size 2.5KB) and flush immediately – in middle am doing compaction at regular intervals. Rate of flushes are around 20flushes/sec. After some time, my RS aborted and never came up back
      with the following error

      2018-08-29 11:34:34,183 DEBUG [flush-table-TestTable_client_1258196-thread-1] regionserver.RSRpcServices: Closing region operation on TestTable_client_1,32816,1535513244999.762a3e633b03e5f847f357aca28768d0.
      2018-08-29 11:34:34,183 INFO  [RpcServer.FifoWFPBQ.default.handler=49,queue=4,port=16040] regionserver.RSRpcServices: flush table task succeed 1, failed 10.
      2018-08-29 11:34:34,280 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=230, memsize=4.2 K, hasBloomFilter=false, into tmp file hdfs://hacluster/hbase/data/hbase/meta/1588230740/.tmp/1cf1deee293848b0bea08940696dbd2a
      2018-08-29 11:34:34,290 INFO  [MemStoreFlusher.0] regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1cf1deee293848b0bea08940696dbd2a
      2018-08-29 11:34:34,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://hacluster/hbase/data/hbase/meta/1588230740/.tmp/1cf1deee293848b0bea08940696dbd2a as hdfs://hacluster/hbase/data/hbase/meta/1588230740/info/1cf1deee293848b0bea08940696dbd2a
      2018-08-29 11:34:34,304 INFO  [MemStoreFlusher.0] regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1cf1deee293848b0bea08940696dbd2a
      2018-08-29 11:34:34,304 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://hacluster/hbase/data/hbase/meta/1588230740/info/1cf1deee293848b0bea08940696dbd2a, entries=13, sequenceid=230, filesize=6.6 K
      2018-08-29 11:34:34,307 FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region server host-xxxx,16040,1535454741321: Replay of WAL required. Forcing server shutdown
      org.apache.hadoop.hbase.DroppedSnapshotException: region: hbase:meta,,1
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2578)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2255)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2217)
              at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2108)
              at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2034)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:505)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:475)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:263)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NullPointerException
              at java.util.ArrayList.<init>(ArrayList.java:177)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:826)
              at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1117)
              at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1090)
              at org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:120)
              at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2450)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2533)
              ... 9 more
      2018-08-29 11:34:34,307 FATAL [MemStoreFlusher.0] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              chandrasekharn Chandra Sekhar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: