Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
2.0.1
-
None
-
None
-
None
-
Mac OS X 10.11.6
Description
It seems that the default warehouse location in Spark 2.0.1 not only points at an inexistent folder in Macintosh systems (/user/hive/warehouse) - see first INFO - but also such folder is then appended to an HDFS - see the error.
This was fixed in 2.0.0, as previous issues have reported, but appears again in 2.0.1. Indeed some scripts I was able to run in 2.0.0 now throw such errors: Spark 2.0.0 used to create the spark-warehouse folder within the current directory (which was good) and didn't complain about such weird paths, even because I'm not using Spark though HDFS, but just locally.
16/10/13 20:47:36 INFO internal.SharedState: Warehouse path is '/user/hive/warehouse'.
py4j.protocol.Py4JJavaError: An error occurred while calling o32.load.
: org.apache.spark.SparkException: Unable to create database default as failed to create its directory hdfs://localhost:9000/user/hive/warehouse
Update #1:
I was able to reinstall Spark 2.0.0 and the first INFO message clearly states that
16/10/13 21:06:59 INFO internal.SharedState: Warehouse path is 'file:/<local FS folder>/spark-warehouse'.
Update #2:
In both Spark 2.0.0 and 2.0.1 I didn't edit any config file and the like. Everything's default.
Attachments
Issue Links
- duplicates
-
SPARK-17810 Default spark.sql.warehouse.dir is relative to local FS but can resolve as HDFS path
- Resolved