Details
-
Bug
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
3.2.1
-
None
-
Spark 3.2.1
Mesos
Java 8 OpenJDK
Description
When updating Spark 3.0.2 to Spark 3.2.1 I get the following error:
Exception in thread "main" org.apache.spark.SparkException: Driver cores must be a positive number
at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257)
at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026)
at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Obviously I set the driver-cores property but it doesn't matter, it is not used. I'm suspecting that it is something about the Mesos Dispatcher, it receives the property because it is being displayed correctly in the Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and therefore fails.
Update:
It happens when submitting a Spark 3.2.1 job with Mesos in Cluster mode.
