Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-1432

JVM hangs on flushing events after OOM

    XMLWordPrintableJSON

Details

    Description

      A hang was observed for ingestion flows after an OOM.

      This appears to be due to the kafka pusher flush() call not returning.

      Hanging stack.

              at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)         at org.apache.gobblin.metrics.kafka.KafkaProducerPusher.flush(KafkaProducerPusher.java:125)         at org.apache.gobblin.metrics.kafka.KafkaProducerPusher.close(KafkaProducerPusher.java:137)         at com.google.common.io.Closer.close(Closer.java:214)         at org.apache.gobblin.metrics.reporter.ConfiguredScheduledReporter.close(ConfiguredScheduledReporter.java:245)         at com.google.common.io.Closer.close(Closer.java:214)         at org.apache.gobblin.metrics.MetricContext.close(MetricContext.java:553)         at org.apache.gobblin.metrics.RootMetricContext$1.run(RootMetricContext.java:194)

       

      Shutdown hook call

      "DestroyJavaVM" #308 prio=5 os_prio=0 tid=0x00007fb55c01a800 nid=0x5d51 in Object.wait() [0x00007fb563f5b000]    java.lang.Thread.State: WAITING (on object monitor)         at java.lang.Object.wait(Native Method)         - waiting on <0x00000000e036cf30> (a org.apache.gobblin.metrics.RootMetricContext$1)         at java.lang.Thread.join(Thread.java:1252)         - locked <0x00000000e036cf30> (a org.apache.gobblin.metrics.RootMetricContext$1)         at java.lang.Thread.join(Thread.java:1326)         at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)         at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)

      Attachments

        Activity

          People

            shirshanka Shirshanka Das
            hanghangliu Hanghang Liu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 50m
                50m