Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44124 Upgrade AWS SDK to v2
  3. SPARK-45721

Upgrade AWS SDK to v2 for Hadoop dependency

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 3.5.0
    • None
    • Build
    • None

    Description

      Hadoop is planning to ship the SDKv2 upgrade in 3.4.0, as shown in this HADOOP-18073. One of the Hadoop modules that Spark relies on is hadoop-aws, which comes with S3A connector that allows Spark to access data in S3 buckets. Hadoop-aws contains dependency on AWS SDKv1, thus we should also update the Hadoop version to 3.4.0.

      Attachments

        Issue Links

          is superceded by

          Sub-task - The sub-task of the issue SPARK-45393 Upgrade Hadoop to 3.4.0

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              cltlfcjin Lantao Jin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: