Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21069

NPE in StoreScanner.updateReaders causes RS to crash

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.2
    • 1.3.3, 1.4.7
    • None
    • None
    • Reviewed

    Description

      I see the following NPE in the region server log for a table that is taking heavy writes.
      I am not sure how the memStoreScanners variable gets set to null.

      2018-08-17 19:59:23,682 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem - Committing store file ...
      2018-08-17 19:59:23,684 INFO  [MemStoreFlusher.1] regionserver.HStore - Added hdfs://...., entries=919170, sequenceid=275114, filesize=22.6 M
      2018-08-17 19:59:23,689 FATAL [MemStoreFlusher.1] regionserver.HRegionServer - ABORTING region server iotperf1dchbase1a-dnds22-2-prd.eng.sfdc.net,60020,1533915690501: Replay of WAL required. Forcing server shutdown
      org.apache.hadoop.hbase.DroppedSnapshotException: region: ......
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2581)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2258)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2220)
              at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2106)
              at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2031)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:508)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:478)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NullPointerException
              at java.util.ArrayList.<init>(ArrayList.java:177)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:827)
              at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1160)
              at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1133)
              at org.apache.hadoop.hbase.regionserver.HStore.access$900(HStore.java:120)
              at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2487)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2536)
              ... 9 more
      2018-08-17 19:59:23,692 FATAL [MemStoreFlusher.1] regionserver.HRegionServer - RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.phoenix.coprocessor.ScanRegionObserver, org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver, org.apache.phoenix.hbase.index.Indexer, org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver, org.apache.hadoop.hbase.security.token.TokenProvider, org.apache.phoenix.coprocessor.ServerCachingEndpointImpl]
      

      Attachments

        1. HBASE-21069-branch-1.patch
          2 kB
          Andrew Kyle Purtell
        2. HBASE-21069-branch-1.patch
          2 kB
          Andrew Kyle Purtell

        Issue Links

          Activity

            People

              apurtell Andrew Kyle Purtell
              tdsilva Thomas D'Silva
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: