Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.0
-
None
Description
In my change to fix SPARK-22372, I removed some code that allowed user customizations of hadoop configs to take effect. Notably, these lines in YarnSparkHadoopUtil:
override def newConfiguration(conf: SparkConf): Configuration = { val hadoopConf = new YarnConfiguration(super.newConfiguration(conf)) hadoopConf.addResource(Client.SPARK_HADOOP_CONF_FILE) hadoopConf }
The addResource call is now missing so user configs are ignored (only the NM config is used).