Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-3713

Archival fails w/ NPE with multi-writers

    XMLWordPrintableJSON

Details

    Description

      With multiple writers, archival runs into NPE. 

       

      22/03/23 11:30:19 INFO TransactionManager: Transaction started without a transaction owner
      22/03/23 11:30:19 ERROR DagScheduler: testjob1 Exception executing node
      java.lang.NullPointerException
              at org.apache.hudi.client.HoodieTimelineArchiver.lambda$getInstantsToArchive$10(HoodieTimelineArchiver.
      java:452)
              at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
              at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
              at java.util.stream.SliceOps$1$1.accept(SliceOps.java:204)
              at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
              at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
              at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
              at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
              at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
              at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
              at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:313
      )
              at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743)
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
              at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
             at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:313
      )
              at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743)
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
              at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
              at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
              at org.apache.hudi.client.HoodieTimelineArchiver.archiveIfRequired(HoodieTimelineArchiver.java:147)
              at org.apache.hudi.client.BaseHoodieWriteClient.archive(BaseHoodieWriteClient.java:818)
              at org.apache.hudi.client.BaseHoodieWriteClient.autoArchiveOnCommit(BaseHoodieWriteClient.java:572)
              at org.apache.hudi.client.BaseHoodieWriteClient.postCommit(BaseHoodieWriteClient.java:477)
              at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:212)
              at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:119)
              at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:615)
              at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)
              at org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper.upsert(HoodieDeltaStreamerWrapper.java:44)
              at org.apache.hudi.integ.testsuite.HoodieTestSuiteWriter.upsert(HoodieTestSuiteWriter.java:163)
              at org.apache.hudi.integ.testsuite.dag.nodes.DeleteNode.ingest(DeleteNode.java:50)
              at org.apache.hudi.integ.testsuite.dag.nodes.InsertNode.execute(InsertNode.java:53)
              at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.executeNode(DagScheduler.java:147)
              at org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler.lambda$execute$0(DagScheduler.java:113)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 

      Attachments

        Issue Links

          Activity

            People

              shivnarayan sivabalan narayanan
              shivnarayan sivabalan narayanan
              Shiyan Xu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: