Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.7.0
-
None
-
None
-
Mac OSX, Spark 2.0.2 and 2.1.0 and Built in Spark
Description
I have included my custom python library in spark submit --files option. It works in zeppelin 0.6.2 but fails in 0.7.0 with error module not found.
Code in notebook
%pyspark
from learney import xyz
from learney import abc
In zeppelin-env.sh (0.6.2 which works)
export SPARK_HOME=/Users/shishircc/old-spark-2.0.2-bin-hadoop2.7
export SPARK_SUBMIT_OPTIONS=" --files /Users/shishircc/learney-pyspark/learney.zip"
In zeppelin-env.sh (0.7.0 which does not work)
export SPARK_HOME=/Users/shishircc/old-spark-2.0.2-bin-hadoop2.7
export SPARK_SUBMIT_OPTIONS=" --files /Users/shishircc/learney-pyspark/learney.zip "
In zeppelin-env.sh (0.7.0 which does not work)
export SPARK_HOME=/Users/shishircc/old-spark-2.1.0-bin-hadoop2.7
export SPARK_SUBMIT_OPTIONS=" --files /Users/shishircc/learney-pyspark/learney.zip "