Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-5741

unable to run spark job within Zeppelin 0.10.1

    XMLWordPrintableJSON

Details

    Description

      Hi,

      I downloaded the Zeppelin docker file from the below site and then i took the spark standalone docker file and merged them into one (attached). I was able to create the image and run the cluster.  

      1) Zeppelin 0.10.1 : https://hub.docker.com/r/apache/zeppelin/dockerfile 
      2) Spark standalone : https://github.com/apache/zeppelin/blob/master/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile

      spark version :

      ENV SPARK_PROFILE="2.4"
      ENV SPARK_VERSION="2.4.8"
      ENV HADOOP_PROFILE="2.7"
      ENV SPARK_HOME="/usr/local/spark"

      Command used to run the container :

      docker run -it \
      -p 8080:8080 \
      -p 7077:7077 \
      -p 8888:8888 \
      -p 8081:8081 \
      -h sparkmaster \
      --name imagename\
      imagename bash;

      3) I am able to connect to Zeppelin Ui : http://hostname:8080/and run basic r/ python notebook but when I run a simple spark code it keeps running and does not finish & its gets stuck. I have restart zep-daemeon everytime to test it again. The application id doesn't even show in the spark master ui: http://hostname:8081/. 

      4) I am able to connect to spark-shell, pyspark and do spark-submit within the container and can see the application in the spark master url but none works from Zeppelin. I have been struggling to make it run from past 1 week

      Note : I had set the spark_home : /usr/local/spark/ and master : spark://sparkmaster:7077 in the interpreter setting for spark but it didn't help. There is also no interpreter log being generated within the zeppelin logs folder.

      I have attached the logs (spark master , slave and zepp logs) but no errors in it. I have pasted the process id details as well. There are many process running which don't get killed when i restart the daemon.

      Attachments

        1. spark.jpg
          108 kB
          Arjun J
        2. Zepp-Spark.jpg
          69 kB
          Arjun J
        3. zepp log.txt
          13 kB
          Arjun J
        4. spark master log.txt
          2 kB
          Arjun J
        5. spark slave log.txt
          2 kB
          Arjun J
        6. Docker merged.txt
          5 kB
          Arjun J

        Activity

          People

            Unassigned Unassigned
            arj.rays Arjun J
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: