Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.0
-
None
Description
Currently, spark-submit --archives and spark.yarn.dist.archives configuration are only supported in Yarn modes:
spark-submit --help
...
Spark on YARN only:
--queue QUEUE_NAME The YARN queue to submit to (Default: "default").
--archives ARCHIVES Comma separated list of archives to be extracted into the
working directory of each executor.
This is actually critical for PySpark to support shipping other packages together, see also https://hyukjin-spark.readthedocs.io/en/stable/user_guide/python_packaging.html#using-zipped-virtual-environment.
Due to this missing feature, PySpark cannot support conda env to ship other packages together.
Attachments
Issue Links
- is related to
-
SPARK-33615 Make spark.archives working in Kubernates
- Resolved
- links to