Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18886

S3A: AWS SDK V2 Migration: stabilization and S3Express

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • fs/s3
    • None

    Description

      The final stabilisation changes to the V2 SDK MIgration; those moved off the HADOOP-18073 JIRA so we can close that.

      also adds support to Amazon S3 Express One Zone storage

      Attachments

        Issue Links

          depends upon

          Bug - A problem which impairs or prevents the functions of the product. HBASE-28056 [HBoss] add support for AWS v2 SDK

          • Major - Major loss of function.
          • In Progress

          Task - A task that needs to be done. HADOOP-18073 S3A: Upgrade AWS SDK to V2

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-18995 S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-44124 Upgrade AWS SDK to v2

          • Major - Major loss of function.
          • Open
          1.
          		 AWS SDK V2 - Refactor getS3Region & other follow up items Sub-task Open Unassigned
          2.
          		 AWS SDK V2 - sigv2 support Sub-task Resolved Unassigned
          3.
          		 AWS v2 SDK: fail meaningfully if legacy v2 signing requested Sub-task Resolved Steve Loughran
          4.
          		 S3ARetryPolicy to use sdk exception retryable() if it is valid Sub-task Resolved Unassigned
          5.
          		 S3A: Cut S3 Select Sub-task Resolved Steve Loughran
          6.
          		 AWS SDK V2 - Implement CSE Sub-task Open Syed Shameerur Rahman
          7.
          		 remove head bucket request from calls which can be made without audit header Sub-task Open Unassigned
          8.
          		 AWS SDK V2 - Move to S3 Java async client Sub-task Open Ahmar Suhail
          9.
          		 Enable dual-layer server-side encryption with AWS KMS keys (DSSE-KMS) Sub-task Resolved Viraj Jasani
          10.
          		 AWS SDK V2 - Add socket factory to Netty Client Sub-task Open Unassigned
          11.
          		 build/release a shaded version of the AWS SDK Sub-task Resolved Unassigned
          12.
          		 S3A. createS3AsyncClient() always enables multipart Sub-task Resolved Steve Loughran
          13.
          		 S3A: V2 SDK client does not work with third-party store Sub-task Resolved Steve Loughran
          14.
          		 Improve s3a region handling, including determining from endpoint Sub-task Resolved Ahmar Suhail
          15.
          		 Tune/extend S3A http connection and thread pool settings Sub-task Resolved Steve Loughran
          16.
          		 ITestS3AHugeFilesEncryption failure Sub-task Resolved Unassigned
          17.
          		 S3ARetryHandler to treat SocketExceptions as connectivity failures Sub-task Resolved Steve Loughran
          18.
          		 Upgrade AWS v2 SDK to 2.20.160 and v1 to 1.12.565 Sub-task Resolved Steve Loughran
          19.
          		 coalesce AWS S3 client proxy settings Sub-task Open Unassigned
          20.
          		 S3A region logic to handle vpce and non standard endpoints Sub-task Resolved Shintaro Onuma
          21.
          		 NPE in AWS v2 SDK RetryOnErrorCodeCondition.shouldRetry() Sub-task Resolved Steve Loughran
          22.
          		 S3A: IAMInstanceCredentialsProvider failing: Failed to load credentials from IMDS Sub-task Resolved Steve Loughran
          23.
          		 S3A: testMultiObjectExceptionFilledIn() assertion error Sub-task Resolved Steve Loughran
          24.
          		 S3A. Add option fs.s3a.directory.operations.purge.uploads to purge on rename/delete Sub-task Resolved Steve Loughran
          25.
          		 S3A third party: document "Certificate doesn't match" Sub-task Open Unassigned
          26.
          		 AWS SDK v2: add path capability probe "fs.s3a.capability.aws.v2" Sub-task Resolved Steve Loughran
          27.
          		 S3A: add s3guard command "bucket" to create buckets Sub-task Resolved Steve Loughran
          28.
          		 s3a file rename does double HEAD or LIST on source file/dir Sub-task Open Unassigned
          29.
          		 AWS SDK v2: extend support for FIPS endpoints Sub-task Resolved Steve Loughran
          30.
          		 S3A credential provider remapping: make extensible Sub-task Resolved Viraj Jasani
          31.
          		 S3A: remove @deprecated tags where no longer needed Sub-task Open Unassigned
          32.
          		 S3A: Support dynamic region resolution Sub-task Open Steve Loughran
          33.
          		 S3A: retry on credential expiry Sub-task Open Unassigned
          34.
          		 S3A: Add option fs.s3a.s3express.create.session to enable/disable CreateSession Sub-task Resolved Steve Loughran
          35.
          		 S3A: debug logging for http traffic to S3 stores Sub-task Resolved Steve Loughran
          36.
          		 S3A to provide full support for S3 Express One Zone Sub-task Resolved Steve Loughran
          37.
          		 Unknown S3Express bucket raises UnknownHostException rather than NoSuchBucketException; will block for retries Sub-task Open Unassigned
          38.
          		 S3A Assume role tests failing against S3Express stores Sub-task Open Unassigned
          39.
          		 S3A: Support Authentication through HttpSigner API Sub-task Resolved Harshit Gupta
          40.
          		 S3A: transfer manager not wired up to s3a executor pool Sub-task Open Unassigned
          41.
          		 S3A: Upgrade AWS SDK to 2.21.41 Sub-task Resolved Steve Loughran
          42.
          		 Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool Sub-task Resolved Mukund Thakur
          43.
          		 S3A : ITestS3AConfiguration#testRequestTimeout failure Sub-task Resolved Steve Loughran
          44.
          		 S3A : ITestS3AConcurrentOps#testParallelRename intermittent timeout failure Sub-task Open Unassigned
          45.
          		 S3A: TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate failure Sub-task Open Unassigned
          46.
          		 S3A: S3AInputStream doesn't recover from HTTP/channel exceptions Sub-task Resolved Steve Loughran
          47.
          		 S3A: disable checksums when fs.s3a.checksum.validation = false Sub-task Resolved Steve Loughran
          48.
          		 S3A: S3A: ITestS3AConfiguration failing with region problems Sub-task Open Unassigned
          49.
          		 S3A: detect and recover from SSL ConnectionReset exceptions Sub-task Open Unassigned
          50.
          		 S3A: Regression: ITestS3AOpenCost fails on prefetch test runs Sub-task Resolved Steve Loughran
          51.
          		 AWS SDK V2 - Update S3A region logic Sub-task Resolved Viraj Jasani
          52.
          		 HADOOP-19045. S3A: CreateSession Timeout after 10 seconds Sub-task Resolved Steve Loughran
          53.
          		 S3A: update AWS sdk versions to 2.23.5 and 1.12.599 Sub-task Resolved Steve Loughran
          54.
          		 S3A: ITestCustomSigner failing against S3Express Buckets Sub-task Open Unassigned
          55.
          		 S3 public test bucket landsat-pds unreadable -needs replacement Sub-task Resolved Steve Loughran
          56.
          		 AWS SDK V2 - Enabling FIPS should be allowed with central endpoint Sub-task Resolved Viraj Jasani
          57.
          		 S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir Sub-task Resolved Viraj Jasani
          58.
          		 provide hadoop binary tarball without aws v2 sdk Sub-task Open Steve Loughran
          59.
          		 core-default fs.s3a.connection.establish.timeout value too low -warning always printed Sub-task Resolved Steve Loughran
          60.
          		 S3A HeaderProcessing to process all metadata entries of HEAD response Sub-task Open Unassigned
          61.
          		 S3 Express: document use Sub-task Open Unassigned
          62.
          		 S3A: handle alternative forms of connection failure Sub-task Open Unassigned
          63.
          		 IAMCredentialsProvider throttling results in AWS auth failures Sub-task Open Unassigned
          64.
          		 S3A initialization/close slower than with v1 SDK Sub-task Resolved Steve Loughran
          65.
          		 s3a: TestS3AAWSCredentialsProvider and TestS3AInputStreamRetry really slow Sub-task Resolved Steve Loughran
          66.
          		 S3A: Unable to recover from failure of multipart block upload attempt "Status Code: 400; Error Code: RequestTimeout" Sub-task Resolved Steve Loughran
          67.
          		 NoSuchMethodError in aws sdk third party logger in hadoop aws 3.4 Sub-task Open Unassigned
          68.
          		 S3ABlockOutputStream no longer sends progress events in close() Sub-task Resolved Steve Loughran

          Activity

            People

              ahmar#1 ahmar#1
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: