Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7606

Design SparkSession, SparkSessionManager [Spark Branch]

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • Spark
    • None

    Description

      In this JIRA we'll design two interfaces:

      • SparkSessionState
      • SparkSessionPoolManager

      and then once that is agreed upon we'll design two implementations:

      • SparkSessionStateImpl
      • SparkSessionPoolManagerImpl

      the form and function of these will be similar to the Tez equivalents. However, TezSessionState provides some implementation which SparkClient already provides (refreshLocalResources*). Let's keep SparkSessionState lightweight and not remove functionality from SparkClient. The scope of this jira is just to create the shells and basic functionality. The implementations in this jira should be able to:

      • Share a SparkSessionImpl across queries
      • Defining when a session can be re-used
      • Take ownership of SparkContext objects (Note we can only have a single SC until SPARK-2243 is resolved)

      Attachments

        1. HIVE-7606.2-spark.patch
          26 kB
          Venki Korukanti
        2. HIVE-7606.1-spark.patch
          25 kB
          Venki Korukanti

        Issue Links

          Activity

            People

              vkorukanti Venki Korukanti
              brocknoland Brock Noland
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: