Uploaded image for project: 'TOREE'
  1. TOREE
  2. TOREE-390

Lazily start Spark sessions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.2.0
    • None
    • None

    Description

      In our deployment, more than half of the startup time for a Toree notebook is taken by starting a Spark session and waiting for containers. Lazily starting Spark sessions helps the notebook environment feel faster, even if the user is waiting on Spark to start up because the time waiting for Spark to start is clearly Spark, not Toree, and is initiated by the user.

      Also, lazily starting a Spark session allows users to change settings that can't be changed in a Spark context. It also enables the same startup code that would be used in a spark-submit application:

      SparkSession.builder
          .config(...)
          .setAppName(...)
          .getOrCreate()
      

      Attachments

        Issue Links

          Activity

            People

              rdblue Ryan Blue
              rdblue Ryan Blue
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: