Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-15696

Spark tests fail with CAMEL_SPARK_HIVE_TESTS=true

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.5.0
    • 3.12.0
    • camel-spark
    • None
    • Unknown

    Description

      Steps to reproduce:

      git checkout camel-3.5.0
      cd components/spark
      export CAMEL_SPARK_HIVE_TESTS=true
      mvn clean test
      ...
      [ERROR] org.apache.camel.component.spark.SparkProducerTest.shouldExecuteConditionalFrameCount  Time elapsed: 0.012 s  <<< ERROR!
      java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat could not be instantiated
              at java.util.ServiceLoader.fail(ServiceLoader.java:232)
              at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
              at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
              at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
              at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
              at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
              at scala.collection.Iterator$class.foreach(Iterator.scala:742)
              at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
              at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
              at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
              at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:258)
              at scala.collection.TraversableLike$class.filter(TraversableLike.scala:270)
              at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
              at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)
              at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:194)
              at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:392)
              at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:326)
              at org.apache.camel.component.spark.SparkProducerTest.createCamelRegistry(SparkProducerTest.java:80)
              at org.apache.camel.test.junit5.CamelTestSupport.createCamelContext(CamelTestSupport.java:730)
              at org.apache.camel.test.junit5.CamelTestSupport.doSetUp(CamelTestSupport.java:412)
              at org.apache.camel.test.junit5.CamelTestSupport.setUp(CamelTestSupport.java:367)
              at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
              at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
              at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
              at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
              at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:126)
              at org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeEachMethod(TimeoutExtension.java:76)
              at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
              at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
              at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
              at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
              at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
              at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
              at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
              at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
              at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:481)
              at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeBeforeEachMethodAdapter$18(ClassBasedTestDescriptor.java:466)
              at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachMethods$2(TestMethodTestDescriptor.java:169)
              at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$5(TestMethodTestDescriptor.java:197)
              at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
              at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:197)
              at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachMethods(TestMethodTestDescriptor.java:166)
              at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:133)
              at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
              at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
              at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
              at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
              at java.util.ArrayList.forEach(ArrayList.java:1259)
              at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
              at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
              at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
              at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
              at java.util.ArrayList.forEach(ArrayList.java:1259)
              at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
              at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
              at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
              at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
              at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
              at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
              at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
              at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
              at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
              at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
              at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
              at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
              at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
              at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:165)
              at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:127)
              at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
              at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
              at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
              at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
      Caused by: java.lang.NoClassDefFoundError: org/slf4j/bridge/SLF4JBridgeHandler
              at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.<init>(ParquetFileFormat.scala:65)
              at sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
              at java.lang.Class.newInstance(Class.java:442)
              at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
              ... 86 more
      Caused by: java.lang.ClassNotFoundException: org.slf4j.bridge.SLF4JBridgeHandler
              at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
              ... 92 more
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ppalaga Peter Palaga
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: