Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8055 Spark Launcher Improvements
  3. SPARK-11035

Launcher: allow apps to be launched in-process

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.6.0
    • 2.3.0
    • Spark Core
    • None

    Description

      The launcher library is currently restricted to launching apps as child processes. That is fine for a lot of cases, especially if the app is running in client mode.

      But in certain cases, especially launching in cluster mode, it's more efficient to avoid launching a new process, since that process won't be doing much.

      We should add support for launching apps in process, even if restricted to cluster mode at first. This will require some rework of the launch paths to avoid using system properties to propagate configuration.

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22290 Starting second context in same JVM fails to get new Hive delegation token

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-21840 Allow multiple SparkSubmit invocations in same JVM without polluting system properties

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-22372 Make YARN client extend SparkApplication

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-22941 Allow SparkSubmit to throw exceptions instead of exiting / printing errors.

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #15009 [Github] Pull Request #15009 (kishorvpatil)

          Pull request #19591 [Github] Pull Request #19591 (vanzin)

          Activity

            People

              vanzin Marcelo Masiero Vanzin
              vanzin Marcelo Masiero Vanzin
              Votes:
              3 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: