Description
HBASE-28401 had a regression due to which HRegion#close throws NPE while trying to close the memstore within the mapper
Due to this, snapshot based MR jobs have started failing in phoenix.
This is due to the fact that TableSnapshotResultIterator ends up trying to release the read lock twice via HRegion#closeRegionOperation
- TableSnapshotResultIterator's next method calls ScanningResultIterator's next method.
-
- ScanningResultIterator's next tries to close the SnapshotScanner early
- Within SnapshotScanner's close method
-
-
- HRegion#closeRegionOperation released the read lock and was successful
- HRegion#close which threw IOException due to memstore issue (
HBASE-28401) - SnapshotScanner catches the IOException but doesn't set region field to null
-
- TableSnapshotResultIterator's finally block calls ScanningResultIterator's close method.
-
-
- ScanningResultIterator's close is called again
- Since region field wasn't null, HRegion#closeRegionOperation is called again and throws IllegalMonitorStateException while trying to release the read lock
-
-
-
- The IllegalMonitorStateException then causes the whole mapper to fail
-
It doesn't cause failure while doing snapshot reads via HBase (ref HBASE-28743 where same NPE was observed but mapper still passes)
, because the closest equivalent code (RecordReader within TableSnapshotInputFormat) doesn't tries to close the region as part of it's nextKeyValue method.
This is generally much safer because record readers are always closed explicitly (even if mapper's run method fails)
There are 2 improvements that can be done here :
1. Disable mslab for region created within snapshot (by setting hbase.hregion.memstore.mslab.enabled set to false)
2. In TableSnapshotResultIterator - Remove the the SnapshotScanner's close (via ScanningResultIterator) called within next method. It would anyways be closed by the mapper at the end
Attachments
Attachments
Issue Links
- is caused by
-
HBASE-28401 Introduce a close method for memstore for release active segment
- Resolved
- links to