Major Description
In our Production, while investigating Region Server crashes, we found that it is due to Phoenix coproc throwing Null Pointer Exception in IndexRegionObserver.postBatchMutateIndispensably() method.
Below are the logs
2024-02-26 13:52:40,716 ERROR [r.default.FPBQ.Fifo.handler=216,queue=8,port=xxxxx] coprocessor.CoprocessorHost - The coprocessor org.apache.phoenix.hbase.index.IndexRegionObserver threw java.lang.NullPointerExceptionjava.lang.NullPointerExceptionat org.apache.phoenix.hbase.index.IndexRegionObserver.postBatchMutateIndispensably(IndexRegionObserver.java:1301)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1028)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1025)at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1025)at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:4134)at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4573)at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4447)at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4369)at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033)at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:951)at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:916)at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) 2024-02-26 13:52:40,725 ERROR [r.default.FPBQ.Fifo.handler=216,queue=8,port=xxxxx] regionserver.HRegionServer - ***** ABORTING region server ssss,xxxxx,1708268161243: The coprocessor org.apache.phoenix.hbase.index.IndexRegionObserver threw java.lang.NullPointerException *****java.lang.NullPointerExceptionat org.apache.phoenix.hbase.index.IndexRegionObserver.postBatchMutateIndispensably(IndexRegionObserver.java:1301)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1028)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1025)at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:558)at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:631)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1025)at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:4134)at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4573)at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4447)at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4369)at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033)at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:951)at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:916)at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)