Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.0
-
None
Description
For SPARK-11035 to work well, at least in cluster mode, YARN needs to implement SparkApplication so that it doesn't use system properties to propagate Spark configuration from spark-submit.
There is a second complication, that YARN uses system properties to propagate SPARK_YARN_MODE on top of other Spark configs. We should take a look at either change that to a configuration, or remove SPARK_YARN_MODE altogether if possible.
Attachments
Issue Links
- relates to
-
SPARK-11035 Launcher: allow apps to be launched in-process
- Resolved
- links to