Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
On some distros/platforms, deploying Livy fails with the following messages.
Error: Systemd start for livy-server failed! journalctl log for livy-server: -- Logs begin at Thu 2020-12-03 09:48:16 UTC, end at Thu 2020-12-03 10:01:07 UTC. -- Dec 03 10:00:59 59fa1cab214f systemd[1]: Starting LSB: Livy Server... Dec 03 10:00:59 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session opened for user livy by (uid=0) Dec 03 10:00:59 59fa1cab214f livy-server[4270]: starting /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.el8_2.aarch64/bin/java -cp /usr/lib/livy/jars/*:/usr/lib/livy/conf:/etc/hadoop/conf: org.apache.livy.server.LivyServer, logging to /var/log/livy/livy-livy-server.out Dec 03 10:01:01 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session closed for user livy Dec 03 10:01:06 59fa1cab214f livy-server[4270]: Failed to start Livy Server. Return value: 1[FAILED] Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Control process exited, code=exited status=1 Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Failed with result 'exit-code'. Dec 03 10:01:06 59fa1cab214f systemd[1]: Failed to start LSB: Livy Server.
https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-aarch64-deploy/2/console
https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-x86_64-deploy/2/console
https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-aarch64-deploy/2/console
https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-x86_64-deploy/2/console
So I reproduced it locally and found the following error in /var/log/livy/livy-livy-server.out, referred to above:
$ docker exec -it 20201204_103251_r9202_bigtop_1 cat /var/log/livy/livy-livy-server.out 20/12/04 10:40:51 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: * Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: SPARK_HOME path does not exist at scala.Predef$.require(Predef.scala:224) at org.apache.livy.utils.LivySparkUtils$.testSparkHome(LivySparkUtils.scala:56) at org.apache.livy.server.LivyServer.start(LivyServer.scala:77) at org.apache.livy.server.LivyServer$.main(LivyServer.scala:423) at org.apache.livy.server.LivyServer.main(LivyServer.scala)
To avoid this situation, Livy server should be started after the spark-core package is installed.