Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
A hang was observed for ingestion flows after an OOM.
This appears to be due to the kafka pusher flush() call not returning.
Hanging stack.
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) at org.apache.gobblin.metrics.kafka.KafkaProducerPusher.flush(KafkaProducerPusher.java:125) at org.apache.gobblin.metrics.kafka.KafkaProducerPusher.close(KafkaProducerPusher.java:137) at com.google.common.io.Closer.close(Closer.java:214) at org.apache.gobblin.metrics.reporter.ConfiguredScheduledReporter.close(ConfiguredScheduledReporter.java:245) at com.google.common.io.Closer.close(Closer.java:214) at org.apache.gobblin.metrics.MetricContext.close(MetricContext.java:553) at org.apache.gobblin.metrics.RootMetricContext$1.run(RootMetricContext.java:194)
Shutdown hook call
"DestroyJavaVM" #308 prio=5 os_prio=0 tid=0x00007fb55c01a800 nid=0x5d51 in Object.wait() [0x00007fb563f5b000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000e036cf30> (a org.apache.gobblin.metrics.RootMetricContext$1) at java.lang.Thread.join(Thread.java:1252) - locked <0x00000000e036cf30> (a org.apache.gobblin.metrics.RootMetricContext$1) at java.lang.Thread.join(Thread.java:1326) at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107) at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)