Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18201

Remove base and bucket overrides for endpoint in ITestS3ARequesterPays.java

    XMLWordPrintableJSON

Details

    Description

      If a user has set the endpoint for their bucket in a test environment, we should ignore that in a test that won't use the bucket that the endpoint is set for. In this case, we are using "s3a://usgs-landsat/" which is in the region us-west-2, and would fail if the user has explicitly set the endpoint to something else.

      Example (I have set the endpoint to ap-south-1):

      [ERROR] testRequesterPaysDisabledFails(org.apache.hadoop.fs.s3a.ITestS3ARequesterPays)  Time elapsed: 9.323 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSRedirectException: getFileStatus on s3a://usgs-landsat/collection02/catalog.json: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: us-west-2. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: Z09V8PMEEN5PHDRZ; S3 Extended Request ID: B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=; Proxy: null), S3 Extended Request ID: B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=:301 Moved Permanently: The bucket is in this region: us-west-2. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: Z09V8PMEEN5PHDRZ; S3 Extended Request ID: B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=; Proxy: null)
 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:233)
 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3440)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3346)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:4890)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$executeOpen$6(S3AFileSystem.java:1437)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.executeOpen(S3AFileSystem.java:1435)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1409)
  
<......>

      CC: stevel@apache.org  mthakur  dannycjones

      Attachments

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue HADOOP-14661 S3A to support Requester Pays Buckets

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-18194 Public dataset class for S3A integration tests

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link GitHub Pull Request #4169

          Activity

            People

              dannycjones Daniel Carl Jones
              mehakmeetSingh Mehakmeet Singh
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m