Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9449

Push YARN configuration to Spark while deply Spark on YARN [Spark Branch]

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • Spark
    • None

    Description

      We only push Spark configuration and RSC configuration to Spark while launch Spark cluster now, for Spark on YARN mode, Spark need extra YARN configuration to launch Spark cluster. Besides this, to support dynamically configuration setting for RSC configuration/YARN configuration, we need to recreate SparkSession while RSC configuration/YARN configuration update as well, as they may influence the Spark cluster deployment as well.

      Attachments

        1. HIVE-9449.1-spark.patch
          5 kB
          Chengxiang Li
        2. HIVE-9449.1-spark.patch
          5 kB
          Xuefu Zhang
        3. HIVE-9449.2-spark.patch
          5 kB
          Chengxiang Li
        4. HiveonSparkconfiguration.pdf
          93 kB
          Chengxiang Li

        Issue Links

          Activity

            People

              chengxiang li Chengxiang Li
              chengxiang li Chengxiang Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: