Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
In our deployment, more than half of the startup time for a Toree notebook is taken by starting a Spark session and waiting for containers. Lazily starting Spark sessions helps the notebook environment feel faster, even if the user is waiting on Spark to start up because the time waiting for Spark to start is clearly Spark, not Toree, and is initiated by the user.
Also, lazily starting a Spark session allows users to change settings that can't be changed in a Spark context. It also enables the same startup code that would be used in a spark-submit application:
SparkSession.builder .config(...) .setAppName(...) .getOrCreate()
Attachments
Issue Links
- is related to
-
TOREE-411 Hive tables are not visible from Toree
- Closed
- links to