Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
2.0.0
-
None
-
None
Description
In Spark 1.6, the following command runs fine with both primary and additional python files in hdfs.
/bin/spark-submit --py-files hdfs:///tmp/base.py --master yarn-cluster hdfs:///tmp/pi.py
In Spark 2.0.0, the following command fails:
/bin/spark-submit --py-files hdfs:///tmp/base.py --master yarn --deploy-mode cluster hdfs:///tmp/pi.py
Error:
Launching Python applications through spark-submit is currently only supported for local files: hdfs:///tmp/base.py
Attachments
Issue Links
- duplicates
-
SPARK-17512 Specifying remote files for Python based Spark jobs in Yarn cluster mode not working
- Resolved