Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Duplicate
-
1.3.1
-
None
-
None
Description
I've created a script that will put continuous put one record (size 2.5KB) and flush immediately – in middle am doing compaction at regular intervals. Rate of flushes are around 20flushes/sec. After some time, my RS aborted and never came up back
with the following error
2018-08-29 11:34:34,183 DEBUG [flush-table-TestTable_client_1258196-thread-1] regionserver.RSRpcServices: Closing region operation on TestTable_client_1,32816,1535513244999.762a3e633b03e5f847f357aca28768d0. 2018-08-29 11:34:34,183 INFO [RpcServer.FifoWFPBQ.default.handler=49,queue=4,port=16040] regionserver.RSRpcServices: flush table task succeed 1, failed 10. 2018-08-29 11:34:34,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=230, memsize=4.2 K, hasBloomFilter=false, into tmp file hdfs://hacluster/hbase/data/hbase/meta/1588230740/.tmp/1cf1deee293848b0bea08940696dbd2a 2018-08-29 11:34:34,290 INFO [MemStoreFlusher.0] regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1cf1deee293848b0bea08940696dbd2a 2018-08-29 11:34:34,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://hacluster/hbase/data/hbase/meta/1588230740/.tmp/1cf1deee293848b0bea08940696dbd2a as hdfs://hacluster/hbase/data/hbase/meta/1588230740/info/1cf1deee293848b0bea08940696dbd2a 2018-08-29 11:34:34,304 INFO [MemStoreFlusher.0] regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1cf1deee293848b0bea08940696dbd2a 2018-08-29 11:34:34,304 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://hacluster/hbase/data/hbase/meta/1588230740/info/1cf1deee293848b0bea08940696dbd2a, entries=13, sequenceid=230, filesize=6.6 K 2018-08-29 11:34:34,307 FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region server host-xxxx,16040,1535454741321: Replay of WAL required. Forcing server shutdown org.apache.hadoop.hbase.DroppedSnapshotException: region: hbase:meta,,1 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2578) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2255) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2217) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2108) at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2034) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:505) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:475) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:263) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at java.util.ArrayList.<init>(ArrayList.java:177) at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:826) at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1117) at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1090) at org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:120) at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2450) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2533) ... 9 more 2018-08-29 11:34:34,307 FATAL [MemStoreFlusher.0] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.
Attachments
Issue Links
- is duplicated by
-
HBASE-21069 NPE in StoreScanner.updateReaders causes RS to crash
- Resolved