Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14029

Update Spark version to 2.0.0

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0
    • None

    Description

      There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark to 2.0.0 to benefit those performance improvements.
      To update Spark version to 2.0.0, the following changes are required:

      • Spark API updates:
        • SparkShuffler#call return Iterator instead of Iterable
        • SparkListener -> JavaSparkListener
        • InputMetrics constructor doesn’t accept readMethod
        • Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics return long type instead of integer
      • Dependency upgrade:
        • Jackson: 2.4.2 -> 2.6.5
        • Netty version: 4.0.23.Final -> 4.0.29.Final
        • Scala binary version: 2.10 -> 2.11
        • Scala version: 2.10.4 -> 2.11.8

      Attachments

        1. HIVE-14029.patch
          23 kB
          Ferdinand Xu
        2. HIVE-14029.8.patch
          27 kB
          Ferdinand Xu
        3. HIVE-14029.7.patch
          28 kB
          Ferdinand Xu
        4. HIVE-14029.6.patch
          26 kB
          Ferdinand Xu
        5. HIVE-14029.5.patch
          24 kB
          Ferdinand Xu
        6. HIVE-14029.4.patch
          24 kB
          Ferdinand Xu
        7. HIVE-14029.3.patch
          24 kB
          Ferdinand Xu
        8. HIVE-14029.2.patch
          24 kB
          Ferdinand Xu
        9. HIVE-14029.1.patch
          32 kB
          Ferdinand Xu

        Issue Links

          is duplicated by

          Wish - General wishlist item. HIVE-14777 Add support of Spark-2.0.0 in Hive-2.X.X

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue SPARK-12154 Upgrade to Jersey 2

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-17563 Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. HIVE-14825 Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14919 Improve the performance of Hive on Spark 2.0.0

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #103

          Web Link review board

          Activity

            People

              Ferd Ferdinand Xu
              Ferd Ferdinand Xu
              Votes:
              1 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: