Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16711

S3A bucket existence checks to support v2 API and "no checks at all"

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.1
    • 3.3.0
    • fs/s3
    • Hide
      The probe for an S3 bucket existing now uses the V2 API, which fails fast if the caller lacks access to the bucket.
      The property fs.s3a.bucket.probe can be used to change this. If using a third party library which doesn't support this API call, change it to "1".

      to skip the probe entirely, use "0". This will make filesystem instantiation slightly faster, at a cost of postponing all issues related to bucket existence and client authentication until the filesystem API calls are first used.
      Show
      The probe for an S3 bucket existing now uses the V2 API, which fails fast if the caller lacks access to the bucket. The property fs.s3a.bucket.probe can be used to change this. If using a third party library which doesn't support this API call, change it to "1". to skip the probe entirely, use "0". This will make filesystem instantiation slightly faster, at a cost of postponing all issues related to bucket existence and client authentication until the filesystem API calls are first used.

    Description

      1. in S3AFileSystem.initialize(), we check for the bucket existing with verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't check for auth issues.

      s3. doesBucketExistV2() does at validate credentials, and should be switched to. This will help things fail faster

      2. For performance it would be good to skip verifyBuckets call during S3A filesystem init(). This would save a call to S3 during init method.

      Attachments

        1. HADOOP-16711.prelim.1.patch
          3 kB
          Rajesh Balamohan

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HADOOP-16995 ITestS3AConfiguration proxy tests fail when bucket probes == 0

          • Major - Major loss of function.
          • Resolved
          causes

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17801 No error message reported when bucket doesn't exist in S3AFS

          • Major - Major loss of function.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-15990 S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-24000 S3A: Create Table should fail on invalid AK/SK

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-15990 S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2

          • Major - Major loss of function.
          • Resolved
          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-17454 [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1838

          Web Link GitHub Pull Request #1845

          Activity

            People

              mukund-thakur Mukund Thakur
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: