Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
-
ghx-label-2
Description
Event processing is paused during a global INVALIDATE METADATA.
catalog_.isEventProcessingActive() returns false in this state.
If an AlterTableAddPartition statement is running during that time, we could pass in a null value for the 'partitionToEventId' map in here:
Map<String, Long> partitionToEventId = catalog_.isEventProcessingActive() ?
Maps.newHashMap() : null;
List<Partition> addedHmsPartitions = addHmsPartitionsInTransaction(msClient,
tbl, allHmsPartitionsToAdd, partitionToEventId, ifNotExists);
If the global INVALIDATE METADATA finishes and event processing is back to the ACTIVE state before the AlterTableAddPartition statement runs into addHmsPartitions(), we will have an non-empty 'partitionToEventSubMap' here:
List<NotificationEvent> events = getNextMetastoreEventsIfEnabled(eventId,
event -> AddPartitionEvent.ADD_PARTITION_EVENT_TYPE
.equals(event.getEventType())
&& msTbl.getDbName().equalsIgnoreCase(event.getDbName())
&& msTbl.getTableName().equalsIgnoreCase(event.getTableName()));
Map<Partition, Long> partitionToEventSubMap = Maps.newHashMap();
getPartitionsFromEvent(events, partitionToEventSubMap);
// set the eventId to last one which we received so the we fetch the next
// set of events correctly
if (!events.isEmpty()) {
eventId = events.get(events.size() - 1).getEventId();
}
if (partitionToEventSubMap.isEmpty()) {
// if partitions couldn't be fetched from events, use the one returned by
// add_partitions call above.
addedHmsPartitions.addAll(addedPartitions);
} else {
Preconditions.checkNotNull(partitionToEventId); // <-- This will fail
Then the AlterTableAddPartition statement fails with NullPointerException:
I0104 02:46:32.075830 1010 jni-util.cc:302] 4a4eae34f60ba947:b6b2bcfc00000000] java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:889)
at org.apache.impala.service.CatalogOpExecutor.addHmsPartitions(CatalogOpExecutor.java:5051)
at org.apache.impala.service.CatalogOpExecutor.addHmsPartitionsInTransaction(CatalogOpExecutor.java:5082)
at org.apache.impala.service.CatalogOpExecutor.alterTableAddPartitions(CatalogOpExecutor.java:4388)
at org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1136)
at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:450)
at org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:304)
at org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
at org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:233)
at org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:247)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:303)