Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3462

Deploying Livy fails if the SPARK_HOME directory doesn't exist

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • deployment
    • None

    Description

      On some distros/platforms, deploying Livy fails with the following messages.

      Error: Systemd start for livy-server failed!
      journalctl log for livy-server:
      -- Logs begin at Thu 2020-12-03 09:48:16 UTC, end at Thu 2020-12-03 10:01:07 UTC. --
      Dec 03 10:00:59 59fa1cab214f systemd[1]: Starting LSB: Livy Server...
      Dec 03 10:00:59 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session opened for user livy by (uid=0)
      Dec 03 10:00:59 59fa1cab214f livy-server[4270]: starting /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.el8_2.aarch64/bin/java  -cp /usr/lib/livy/jars/*:/usr/lib/livy/conf:/etc/hadoop/conf: org.apache.livy.server.LivyServer, logging to /var/log/livy/livy-livy-server.out
      Dec 03 10:01:01 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session closed for user livy
      Dec 03 10:01:06 59fa1cab214f livy-server[4270]: Failed to start Livy Server. Return value: 1[FAILED]
      Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Control process exited, code=exited status=1
      Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Failed with result 'exit-code'.
      Dec 03 10:01:06 59fa1cab214f systemd[1]: Failed to start LSB: Livy Server.
      

      https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-aarch64-deploy/2/console
      https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-x86_64-deploy/2/console
      https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-aarch64-deploy/2/console
      https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-x86_64-deploy/2/console

      So I reproduced it locally and found the following error in /var/log/livy/livy-livy-server.out, referred to above:

      $ docker exec -it 20201204_103251_r9202_bigtop_1 cat /var/log/livy/livy-livy-server.out
      20/12/04 10:40:51 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: *
      Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: SPARK_HOME path does not exist
      	at scala.Predef$.require(Predef.scala:224)
      	at org.apache.livy.utils.LivySparkUtils$.testSparkHome(LivySparkUtils.scala:56)
      	at org.apache.livy.server.LivyServer.start(LivyServer.scala:77)
      	at org.apache.livy.server.LivyServer$.main(LivyServer.scala:423)
      	at org.apache.livy.server.LivyServer.main(LivyServer.scala)
      

      To avoid this situation, Livy server should be started after the spark-core package is installed.

      Attachments

        Activity

          People

            sekikn Kengo Seki
            sekikn Kengo Seki
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 10m
                1h 10m