Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33772

Build and Run Spark on Java 17

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • Build

    Description

      Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 17.

      Version Release Date
      Java 17 (LTS) September 2021

      Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along with the release branch cut.

      Supporting new Java version is considered as a new feature which we cannot allow to backport.

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-35781 Support Spark on Apple Silicon on macOS natively on Java 17

          • Major - Major loss of function.
          • Resolved

          Umbrella - An overarching type made of sub-tasks SPARK-38396 Improve K8s Integration Tests

          • Major - Major loss of function.
          • Resolved
          is required by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-12240 Make Beam compatible with next Java LTS version (Java 17)

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Triage Needed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-17177 Java 17 support

          • Major - Major loss of function.
          • Open
          1.
          		 Add Java 17 ea build test to GitHub action Sub-task Resolved William Hyun  
          2.
          		 Update commons-lang3 to 3.11 Sub-task Resolved William Hyun  
          3.
          		 Upgrade checkstyle plugins Sub-task Resolved William Hyun  
          4.
          		 Upgrade ORC to 1.7.1 Sub-task Resolved William Hyun  
          5.
          		 Move Java 17 on GitHub Actions from EA to LTS release Sub-task Resolved Kousuke Saruta  
          6.
          		 Upgrade ZooKeeper to 3.6.2 Sub-task Resolved Yuming Wang  
          7.
          		 Exclude hadoop-client-api transitive dependency from orc-core Sub-task Resolved Kousuke Saruta  
          8.
          		 Change HiveExternalCatalogVersionsSuite to be able to test with Java 17 Sub-task Resolved Kousuke Saruta  
          9.
          		 Fix Invalid value for HourOfAmPm when testing on JDK 14 Sub-task Resolved Yuming Wang  
          10.
          		 Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS Sub-task Resolved Yuming Wang  
          11.
          		 Upgrade Scala to 2.12.14 Sub-task Resolved Dongjoon Hyun  
          12.
          		 Upgrade Scala to 2.12.15 Sub-task Resolved Dongjoon Hyun  
          13.
          		 Upgrade ORC to 1.7.0 Sub-task Resolved Dongjoon Hyun  
          14.
          		 Add Daily GitHub Action jobs for Java11/17 Sub-task Resolved Dongjoon Hyun  
          15.
          		 Remove explicit GC options from Scala tests Sub-task Resolved Dongjoon Hyun  
          16.
          		 Add `.java-version` to `.gitignore` and `.rat-excludes` Sub-task Resolved Dongjoon Hyun  
          17.
          		 Add Java17 PySpark daily test coverage Sub-task Resolved Dongjoon Hyun  
          18.
          		 Add Java17 SparkR daily test coverage Sub-task Resolved Dongjoon Hyun  
          19.
          		 Document Java 17 support Sub-task Resolved Dongjoon Hyun  
          20.
          		 Upgrade Maven to 3.8.4 Sub-task Resolved Dongjoon Hyun  
          21.
          		 Upgrade SBT to 1.6.0 Sub-task Resolved Dongjoon Hyun  
          22.
          		 Set file.encoding to UTF-8 for SBT tests Sub-task Resolved Dongjoon Hyun  
          23.
          		 Use static factory methods instead of the deprecated `Byte/Short/Integer/Long` constructors Sub-task Resolved Dongjoon Hyun  
          24.
          		 Disable to_timestamp('366', 'DD') test case Sub-task Resolved Dongjoon Hyun  
          25.
          		 Upgrade Scala 2.13 to 2.13.7 Sub-task Resolved Yang Jie  
          26.
          		 Make dev/mima runs on Java 17 Sub-task Resolved Yang He  
          27.
          		 Make sql/core and dependent modules all UTs pass on Java 17 Sub-task Resolved Yang Jie  
          28.
          		 "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17 Sub-task Resolved Yang Jie  
          29.
          		 Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior. Sub-task Resolved Yang Jie  
          30.
          		 WholeStageCodegenSparkSubmitSuite Failed with Java 17 Sub-task Resolved Yang Jie  
          31.
          		 `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17 Sub-task Resolved Yang Jie  
          32.
          		 Pass all UTs in `mllib-local` and `mllib` with Java 17 Sub-task Resolved Yang Jie  
          33.
          		 Pass all UTs in `repl` with Java 17 Sub-task Resolved Yang Jie  
          34.
          		 Pass all UTs in `external/avro` with Java 17 Sub-task Resolved Yang Jie  
          35.
          		 Pass all UTs in `sql/hive` with Java 17 Sub-task Resolved Yang Jie  
          36.
          		 Pass all UTs in `yarn` with Java 17 Sub-task Resolved Yang Jie  
          37.
          		 Supplement all micro benchmark results use to Java 17 Sub-task Resolved Yang Jie  
          38.
          		 Upgrade RoaringBitmap to 0.9.23 Sub-task Resolved Yang Jie  
          39.
          		 Add InMemoryColumnarBenchmark and StateStoreBasicOperationsBenchmark Java 11/17 result Sub-task Resolved Yang Jie  
          40.
          		 Try to remove `--add-exports` compile option for Java 17 Sub-task Resolved Yang Jie  
          41.
          		 Add Java 17 support for spark pull request builds Sub-task Closed Unassigned  
          42.
          		 Add a vectorized BLAS implementation Sub-task Resolved Ludovic Henry  
          43.
          		 Support Java 17 in SparkR SystemRequirements Sub-task Resolved Darek

          0%
          Original Estimate - 1h
          Remaining Estimate - 1h
          44.
          		 Support Java 17 in `dev/test-dependencies.sh` Sub-task Closed Unassigned  
          45.
          		 Adapt uses of JDK 17 Internal APIs Sub-task Closed Unassigned  

          Activity

            People

              LuciferYang Yang Jie
              dongjoon Dongjoon Hyun
              Votes:
              3 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 1h
                  1h
                  Remaining:
                  Remaining Estimate - 1h
                  1h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified