Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22372

Make YARN client extend SparkApplication

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.0
    • Spark Core, YARN
    • None

    Description

      For SPARK-11035 to work well, at least in cluster mode, YARN needs to implement SparkApplication so that it doesn't use system properties to propagate Spark configuration from spark-submit.

      There is a second complication, that YARN uses system properties to propagate SPARK_YARN_MODE on top of other Spark configs. We should take a look at either change that to a configuration, or remove SPARK_YARN_MODE altogether if possible.

      Attachments

        Issue Links

          Activity

            People

              vanzin Marcelo Masiero Vanzin
              vanzin Marcelo Masiero Vanzin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: