Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2292

Pyspark interpreter module not found even though --files parameter provided

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.7.0
    • None
    • pySpark
    • None
    • Mac OSX, Spark 2.0.2 and 2.1.0 and Built in Spark

    Description

      I have included my custom python library in spark submit --files option. It works in zeppelin 0.6.2 but fails in 0.7.0 with error module not found.

      Code in notebook
      %pyspark
      from learney import xyz
      from learney import abc

      In zeppelin-env.sh (0.6.2 which works)
      export SPARK_HOME=/Users/shishircc/old-spark-2.0.2-bin-hadoop2.7
      export SPARK_SUBMIT_OPTIONS=" --files /Users/shishircc/learney-pyspark/learney.zip"

      In zeppelin-env.sh (0.7.0 which does not work)
      export SPARK_HOME=/Users/shishircc/old-spark-2.0.2-bin-hadoop2.7
      export SPARK_SUBMIT_OPTIONS=" --files /Users/shishircc/learney-pyspark/learney.zip "

      In zeppelin-env.sh (0.7.0 which does not work)
      export SPARK_HOME=/Users/shishircc/old-spark-2.1.0-bin-hadoop2.7
      export SPARK_SUBMIT_OPTIONS=" --files /Users/shishircc/learney-pyspark/learney.zip "

      Attachments

        Activity

          People

            Unassigned Unassigned
            shishircc@gmail.com Shishir Choudhary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: