Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47046

Apache Spark 4.0.0 Dependency Audit and Cleanup

    XMLWordPrintableJSON

Details

    Attachments

      Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

        • Critical - Crashes, loss of data, severe memory leak.
        • Open
        relates to

        Sub-task - The sub-task of the issue SPARK-47018 Upgrade built-in Hive to 2.3.10

        • Major - Major loss of function.
        • Resolved

        Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-45869 Revisit and Improve Spark Standalone Cluster

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        links to

        Web Link GitHub Pull Request #47200

        1.
        		 Upgrade Derby to 10.15.1.3+ Sub-task Resolved Yang Jie
        2.
        		 Upgrade Derby to 10.16.1.1 Sub-task Resolved Yang Jie
        3.
        		 Add `derby-provided` profile Sub-task Resolved Dongjoon Hyun
        4.
        		 Add `hive-jackson-provided` profile Sub-task Resolved Dongjoon Hyun
        5.
        		 Provide `CodeHaus Jackson` dependencies via a new optional directory Sub-task Resolved Dongjoon Hyun
        6.
        		 Remove Guava from shared classes from IsolatedClientLoader Sub-task Resolved Cheng Pan
        7.
        		 Upgrade `Guava` dependency in `docker-integration-tests` test module Sub-task Resolved Dongjoon Hyun
        8.
        		 Upgrade Guava used by the connect module to 33.1.0-jre Sub-task Resolved Pan Bingkun
        9.
        		 Upgrade ZooKeeper to 3.9.1 Sub-task Resolved Bjørn Jørgensen
        10.
        		 Upgrade `ZooKeeper` to 3.9.2 Sub-task Resolved Dongjoon Hyun
        11.
        		 Upgrade dropwizard metrics 4.2.21 Sub-task Resolved Yang Jie
        12.
        		 Upgrade dropwizard metrics 4.2.25 Sub-task Resolved Pan Bingkun
        13.
        		 Upgrade dropwizard metrics to 4.2.26 Sub-task Resolved Wei Guo
        14.
        		 Upgrade dropwizard metrics to 4.2.27 Sub-task Resolved Yang Jie
        15.
        		 Remove the special Zookeeper version in the `streaming-kafka-0-10` and `sql-kafka-0-10` modules Sub-task Resolved Yang Jie
        16.
        		 Migrating Junit4 to Junit5 Sub-task Resolved Yang Jie
        17.
        		 Upgrade jetty to 9.4.53.v20231009 Sub-task Resolved Yang Jie
        18.
        		 Migrate jetty 9 to jetty 10 Sub-task Resolved HiuFung Kwok
        19.
        		 Upgrade Jetty to 11 Sub-task Resolved HiuFung Kwok
        20.
        		 Remove shaded `protobuf-java` 2.6.1 dependency from `kinesis-asl-assembly` Sub-task Resolved Dongjoon Hyun
        21.
        		 Fix `kvstore` module to have explicit `commons-lang3` test dependency Sub-task Resolved Dongjoon Hyun
        22.
        		 Use `org.seleniumhq.selenium.htmlunit3-driver` instead of `net.sourceforge.htmlunit` Sub-task Resolved Dongjoon Hyun
        23.
        		 Upgrade `aircompressor` to 0.26 Sub-task Resolved Dongjoon Hyun
        24.
        		 Update `aircompressor` to 0.27 Sub-task Resolved Bjørn Jørgensen
        25.
        		 Upgrade `aircompressor` to 2.0.2 Sub-task Resolved Pan Bingkun
        26.
        		 Ban non-shaded Hadoop dependencies Sub-task Resolved Dongjoon Hyun
        27.
        		 Add a checkstyle rule to ban `commons-lang` in Java code Sub-task Resolved Dongjoon Hyun
        28.
        		 Add scalastyle and checkstyle rules to ban `FileBackedOutputStream` Sub-task Resolved Dongjoon Hyun
        29.
        		 When there are corrupted files in the local maven repo, retry to skip this cache Sub-task Resolved Pan Bingkun
        30.
        		 Add `scalastyle` and `checkstyle` rules to ban `AtomicDoubleArray|CompoundOrdering` Sub-task Resolved Dongjoon Hyun
        31.
        		 Upgrade several Maven plugins to the latest versions Sub-task Resolved Dongjoon Hyun
        32.
        		 Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir Sub-task Resolved Dongjoon Hyun
        33.
        		 Migrate mockito 4 to mockito5 Sub-task Resolved Pan Bingkun
        34.
        		 Add `getUriBuilder` to `o.a.s.u.Utils` and use it Sub-task Resolved Dongjoon Hyun
        35.
        		 Upgrade `mockito` to 5.10.0 Sub-task Resolved Dongjoon Hyun
        36.
        		 Upgrade Netty to 4.1.106.Final Sub-task Resolved Pan Bingkun
        37.
        		 Add netty-tcnative-boringssl-static dependency Sub-task Resolved Hasnain Lakhani
        38.
        		 Upgrade netty to 4.1.107.Final and netty-tcnative to 2.0.62.Final Sub-task Resolved Dongjoon Hyun
        39.
        		 Upgrade `netty` to 4.1.108.Final and `netty-tcnative` to 2.0.65.Final Sub-task Resolved Pan Bingkun
        40.
        		 Upgrade netty to `4.1.109.Final` Sub-task Resolved Pan Bingkun
        41.
        		 Upgrade scalatest related dependencies to the 3.2.18 series Sub-task Resolved Yang Jie
        42.
        		 Fix `HistoryServerSuite` to get `getNumJobs` in `eventually` Sub-task Resolved Dongjoon Hyun
        43.
        		 Set `derby.connection.requireAuthentication` to false explicitly in CLIs Sub-task Resolved Dongjoon Hyun
        44.
        		 Upgrade commons-compress.version to 1.24.0 Sub-task Resolved Hyukjin Kwon
        45.
        		 Upgrade commons-compress to 1.25.0 Sub-task Resolved Yang Jie
        46.
        		 Upgrade `commons-compress` to 1.26.0 Sub-task Resolved Dongjoon Hyun
        47.
        		 Upgrade `commons-compress` to 1.26.1` Sub-task Resolved Dongjoon Hyun
        48.
        		 Upgrade `commons-compress` to 1.27.0 Sub-task Resolved Pan Bingkun
        49.
        		 Revert S3A endpoint fixup logic of SPARK-35878 Sub-task Resolved Steve Loughran
        50.
        		 Remove javax-servlet-api exclusion rule for SBT Sub-task Resolved HiuFung Kwok
        51.
        		 Update `SKIP_SPARK_RELEASE_VERSIONS` in Maven CIs Sub-task Resolved Dongjoon Hyun
        52.
        		 Fix `MavenUtilsSuite` to use `MavenUtils.resolveMavenCoordinates` properly Sub-task Resolved Dongjoon Hyun
        53.
        		 Use `spark.jars.ivy` instead `spark.driver.extraJavaOptions` in `DepsTestsSuite` Sub-task Resolved Dongjoon Hyun
        54.
        		 Upgrade Ivy to 2.5.2 Sub-task Resolved Dongjoon Hyun
        55.
        		 Improve `ArtifactSuite` to use unique `MavenCoordinate`s Sub-task Resolved Dongjoon Hyun
        56.
        		 Migrate jersey 2 to jersey 3 Sub-task Resolved Unassigned
        57.
        		 Fix `kafka-0-10-sql` to use `ResetSystemProperties` if `KafkaTestUtils` is used Sub-task Resolved Dongjoon Hyun
        58.
        		 Update K8s `Dockerfile` to include `hive-jackson` directory if exists Sub-task Resolved Dongjoon Hyun
        59.
        		 Fix `make-distribution.sh` to check `jackson-core-asl-1.9.13.jar` existence first Sub-task Resolved Dongjoon Hyun
        60.
        		 Remove redundant scope identifier for `jakarta.servlet-api` and `javax.servlet-api` Sub-task Resolved HiuFung Kwok
        61.
        		 Remove ZOOKEEPER-1844 comment from KafkaTestUtils Sub-task Resolved Dongjoon Hyun
        62.
        		 Exclude `commons-(io|lang3)` transitive dependencies from `commons-compress` and `avro-*` Sub-task Resolved Dongjoon Hyun
        63.
        		 Upgrade log4j2 to 2.23.0 Sub-task Resolved Unassigned
        64.
        		 Upgrade slf4j to 2.0.12 Sub-task Resolved Yang Jie
        65.
        		 Upgrade `slf4j` to 2.0.14 Sub-task Resolved Pan Bingkun
        66.
        		 Bump ap-loader 3.0(v8) to support for async-profiler 3.0 Sub-task Resolved Nicholas Jiang
        67.
        		 Upgrade jetty to 11.0.20 Sub-task Resolved Yang Jie
        68.
        		 Upgrade `grpcio*` and `grpc-java` to 1.62 Sub-task Resolved Yang Jie
        69.
        		 Upgrade rocksdbjni to 8.8.1 Sub-task Resolved Pan Bingkun
        70.
        		 Upgrade rocksdbjni to 8.11.3 Sub-task Resolved Yang Jie
        71.
        		 Upgrade rocksdbjni to 8.11.4 Sub-task Resolved Neil Ramaswamy
        72.
        		 Upgrade `rocksdbjni` to 9.2.1 Sub-task Resolved Pan Bingkun
        73.
        		 Upgrade rocksdbjni to 9.5.2 Sub-task Resolved Pan Bingkun
        74.
        		 Upgrade RoaringBitmap to 1.0.5 Sub-task Resolved Pan Bingkun
        75.
        		 Upgrade `gcs-connector` to 2.2.16 Sub-task Resolved Dongjoon Hyun
        76.
        		 Upgrade `gcs-connector` to 2.2.17 Sub-task Resolved Dongjoon Hyun
        77.
        		 Upgrade `gcs-connector` to 2.2.18 Sub-task Resolved Dongjoon Hyun
        78.
        		 Upgrade `gcs-connector` to 2.2.20 Sub-task Resolved Dongjoon Hyun
        79.
        		 Upgrade `gcs-connector` to 2.2.21 and add a note for 3.0.0 Sub-task Resolved Dongjoon Hyun
        80.
        		 Update code style' plugins to latest version Sub-task Resolved Pan Bingkun
        81.
        		 Upgrade jackson to 2.16.0 Sub-task Resolved Pan Bingkun
        82.
        		 Upgrade jackson to 2.17.0 Sub-task Resolved Pan Bingkun
        83.
        		 Upgrade jackson to 2.17.1 Sub-task Resolved Pan Bingkun
        84.
        		 Upgrade jackson to 2.17.2 Sub-task Resolved Wei Guo
        85.
        		 Upgrade jackson to 2.18.0 Sub-task Resolved Pan Bingkun
        86.
        		 Upgrade jmock-junit5 to 2.13.1 Sub-task Resolved Yang Jie
        87.
        		 Remove `commons-logging` dependency Sub-task Resolved Dongjoon Hyun
        88.
        		 Remove unused `commons-beanutils` dependency Sub-task Resolved Dongjoon Hyun
        89.
        		 Upgrade `sbt-assembly` to `2.2.0` and `sbt-protoc` to `1.0.7` Sub-task Resolved Pan Bingkun
        90.
        		 Upgrade `zstd-jni` to 1.5.6-1 Sub-task Resolved Dongjoon Hyun
        91.
        		 Upgrade `zstd-jni` to 1.5.6-2 Sub-task Resolved Pan Bingkun
        92.
        		 Upgrade `zstd-jni` to 1.5.6-3 Sub-task Resolved Pan Bingkun
        93.
        		 Upgrade `zstd-jni` to 1.5.6-5 Sub-task Resolved Pan Bingkun
        94.
        		 Exclude `org.junit.jupiter` and `org.junit.platform` from `jmock-junit5` Sub-task Resolved Yang Jie
        95.
        		 Upgrade `tink` to 1.12.0 Sub-task Resolved Pan Bingkun
        96.
        		 Upgrade tink to 1.13.0 Sub-task Resolved Yang Jie
        97.
        		 Upgrade tink to 1.14.1 Sub-task Resolved Wei Guo
        98.
        		 Upgrade tink to 1.15.0 Sub-task Resolved Pan Bingkun
        99.
        		 Upgrade snappy to 1.1.10.4 Sub-task Resolved Bjørn Jørgensen
        100.
        		 Upgrade snappy to 1.1.10.5 Sub-task Resolved Pan Bingkun
        101.
        		 Upgrade snappy to 1.1.10.6 Sub-task Resolved Pan Bingkun
        102.
        		 Upgrade snappy to 1.1.10.7 Sub-task Resolved Yang Jie
        103.
        		 Remove redundant json4s-jackson definition in sql/api POM Sub-task Resolved Cheng Pan
        104.
        		 Upgrade `commons-io` to 2.14.0 Sub-task Resolved Pan Bingkun
        105.
        		 Upgrade commons-io to 2.15.0 Sub-task Resolved Yang Jie
        106.
        		 Upgrade commons-io to 2.15.1 Sub-task Resolved Pan Bingkun
        107.
        		 Upgrade commons-io to 2.16.0 Sub-task Resolved Pan Bingkun
        108.
        		 Upgrade `commons-io` to 2.16.1 Sub-task Resolved Dongjoon Hyun
        109.
        		 Upgrade `commons-io` to 2.17.0 Sub-task Resolved Pan Bingkun
        110.
        		 Upgrade `commons-codec` to 1.16.1 Sub-task Resolved Pan Bingkun
        111.
        		 Upgrade `commons-codec` to 1.17.0 Sub-task Resolved Pan Bingkun
        112.
        		 Upgrade `commons-lang3` to 3.15.0 Sub-task Resolved Pan Bingkun
        113.
        		 Upgrade `commons-lang3` to 3.16.0 Sub-task Resolved Pan Bingkun
        114.
        		 Upgrade commons-text to 1.12.0 Sub-task Resolved Yang Jie
        115.
        		 Bump json4s 4.0.7 Sub-task Resolved Cheng Pan
        116.
        		 Upgrade `postgresql` to 42.7.3 Sub-task Resolved Pan Bingkun
        117.
        		 Upgrade `slf4j` to 2.0.13 Sub-task Resolved Dongjoon Hyun
        118.
        		 Upgrade `guava` dependency to `33.1.0-jre` in Docker IT Sub-task Resolved Cheng Pan
        119.
        		 Upgrade built-in Hive to 2.3.10 Sub-task Resolved Cheng Pan
        120.
        		 Upgrade RoaringBitmap to 1.0.6 Sub-task Resolved Pan Bingkun
        121.
        		 Upgrade ASM to 9.7 Sub-task Resolved Pan Bingkun
        122.
        		 Upgrade maven plugin to latest version Sub-task Resolved Pan Bingkun
        123.
        		 Add `commons-lang:commons-lang:2.6` back to support legacy Hive UDF jars Sub-task Resolved Dongjoon Hyun
        124.
        		 Upgrade scalacheck to 1.18.0 Sub-task Resolved Wei Guo
        125.
        		 Upgrade curator to 5.7.0 Sub-task Resolved Wei Guo
        126.
        		 Upgrade RoaringBitmap to 1.1.0 Sub-task Resolved Wei Guo
        127.
        		 Upgrade `scalatest` related dependencies to the 3.2.19 series Sub-task Resolved Wei Guo
        128.
        		 Upgrade `gcs-connector` to `2.2.25` Sub-task Resolved Dongjoon Hyun
        129.
        		 Upgrade mysql-connector-j to 9.0.0 Sub-task Resolved Wei Guo
        130.
        		 Upgrade mariadb-java-client to 3.4.0 Sub-task Resolved Unassigned
        131.
        		 Upgrade docker-java to 3.4.0 Sub-task Resolved Wei Guo
        132.
        		 Upgrade slf4j to 2.0.16 Sub-task Resolved Yang Jie
        133.
        		 Upgrade `xz` to `1.10` Sub-task Resolved Dongjoon Hyun
        134.
        		 Upgrade `netty-tcnative` to 2.0.66.Final Sub-task Resolved Pan Bingkun
        135.
        		 Upgrade joda-time to 2.13.0 Sub-task Resolved Pan Bingkun
        136.
        		 Upgrade `protobuf-java` to `3.25.3` Sub-task Resolved Pan Bingkun
        137.
        		 Upgrade `protobuf-java` to `3.25.4` Sub-task Resolved pengqli
        138.
        		 Upgrade `protobuf-java` to `3.25.5` Sub-task Resolved Dongjoon Hyun
        139.
        		 Upgrade zstd-jni to 1.5.5-11 Sub-task Resolved Dongjoon Hyun
        140.
        		 Upgrade Netty to 4.1.100.Final Sub-task Resolved Dongjoon Hyun
        141.
        		 Upgrade `kubernetes-client` to 6.8.1 Sub-task Resolved Dongjoon Hyun

        Activity

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: