[ZEPPELIN-6030] Bug Report: Apache Zeppelin 0.11.1 Issue with Spark Interpreter - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 0.11.1
Fix Version/s: 0.12.0, 0.11.2
Component/s: build
Labels:
None
Environment:
Hide

Environment:

Apache Zeppelin version: 0.11.1

Operating System: Ubuntu 24.04 LTS

Java version: OpenJDK 8

Spark version: 2.4.7

Hadoop: Fully distributed mode

YARN: Fully distributed mode

Zeppelin server and Spark running on different servers

Spark configured to use YARN in cluster mode
Show
Environment: Apache Zeppelin version: 0.11.1 Operating System: Ubuntu 24.04 LTS Java version: OpenJDK 8 Spark version: 2.4.7 Hadoop: Fully distributed mode YARN: Fully distributed mode Zeppelin server and Spark running on different servers Spark configured to use YARN in cluster mode

Flags:

Important

Description

Description:

While using Apache Zeppelin version 0.11.1, an issue occurs when running the Spark interpreter. The interpreter fails to initialize properly due to unexpected metadata files that were likely included during the build process on a macOS system. These metadata files cause errors in the initialization of the Spark interpreter, preventing it from functioning correctly.

Maybe there's an error in the build.

Steps to Reproduce:

Set up Apache Zeppelin version 0.11.1.
Configure the Spark interpreter.
Attempt to run a Spark paragraph.

Expected Behavior: The Spark paragraph should execute without any errors.

Actual Behavior: The Spark interpreter fails to initialize, and the following error messages are logged:

Here, you can see that the JAR file you want to run starts with ._ and it's actually not a JAR file, it's a metadata file for apple mac.

Caused by: java.io.IOException: Error accessing /tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0006/container_1719905050899_0006_01_000001/._spark-scala-2.12-0.11.1.jar at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:190) at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:238) at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:235) at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:272) at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:272) at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62) at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:58) at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:105) ... 40 more Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:231) at java.util.zip.ZipFile.<init>(ZipFile.java:157) at java.util.zip.ZipFile.<init>(ZipFile.java:171) at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:187) ... 49 more

Additional Information:

The issue is caused by macOS metadata files prefixed with '._' being included in the Spark interpreter directory. (I don't think this metadata file was in the previous version, so I think it went in by mistake.)

References:

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: onjee

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 08/Jul/24 03:27

Updated:: 12/Sep/24 01:10

Resolved:: 12/Sep/24 01:10