Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-7513

Add jackson-module-scala to spark bundle

    XMLWordPrintableJSON

Details

    Description

      When I do spark stream read, get NoClassDefFoundError.

      // code placeholder
      
      24/03/18 14:15:37 ERROR MicroBatchExecution: Query [id = 2f7a09fb-03f2-4b21-bf72-79295daeda5a, runId = 5d627c7a-e469-4f91-bd84-c012e393cd60] terminated with error
      java.lang.NoClassDefFoundError: org/apache/hudi/com/fasterxml/jackson/module/scala/DefaultScalaModule$
          at org.apache.spark.sql.hudi.streaming.HoodieStreamSource.$anonfun$getBatch$1(HoodieStreamSource.scala:142)
          at scala.Option.map(Option.scala:230)
          at org.apache.spark.sql.hudi.streaming.HoodieStreamSource.getBatch(HoodieStreamSource.scala:142)
          at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$populateStartOffsets$4(MicroBatchExecution.scala:354)
          at scala.collection.Iterator.foreach(Iterator.scala:943)
          at scala.collection.Iterator.foreach$(Iterator.scala:943)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
          at scala.collection.IterableLike.foreach(IterableLike.scala:74)
          at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
          at org.apache.spark.sql.execution.streaming.StreamProgress.foreach(StreamProgress.scala:27)
          at org.apache.spark.sql.execution.streaming.MicroBatchExecution.populateStartOffsets(MicroBatchExecution.scala:351)
          at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:224)
          at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
          at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:375)
          at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:373)
          at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
          at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:219)
          at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:67)
          at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:213)
          at org.apache.spark.sql.execution.streaming.StreamExecution.$anonfun$runStream$1(StreamExecution.scala:307)
          at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
          at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
          at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:285)
          at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:208)
      Caused by: java.lang.ClassNotFoundException: org.apache.hudi.com.fasterxml.jackson.module.scala.DefaultScalaModule$
          at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:351) 

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              xichaomin xi chaomin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: