Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3733

"s3:" URLs break when Secret Key contains a slash, even if encoded

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.17.1, 2.0.2-alpha
    • 2.8.0, 3.0.0-alpha1
    • fs/s3
    • None
    • Reviewed
    • Allows userinfo component of URI authority to contain a slash (escaped as %2F). Especially useful for accessing AWS S3 with distcp or hadoop fs.

    Description

      When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails if the SECRET contains a slash, even when the slash is URL-encoded as %2F.

      Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
      And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
      And your bucket is called "mybucket"

      You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv

      But this doesn't work:

      $ bin/hadoop distcp file:///source  s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket
org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden
        at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
...
With failures, global counters are inaccurate; consider running with -i
Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
        at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
...

      Attachments

        1. hadoop-3733.patch
          2 kB
          Paul Butler
        2. HADOOP-3733.patch
          7 kB
          David Chaiken
        3. HADOOP-3733-20130223T011025Z.patch
          7 kB
          David Chaiken
        4. HADOOP-3733-branch-2-001.patch
          31 kB
          Steve Loughran
        5. HADOOP-3733-branch-2-002.patch
          32 kB
          Steve Loughran
        6. HADOOP-3733-branch-2-003.patch
          40 kB
          Steve Loughran
        7. HADOOP-3733-branch-2-004.patch
          40 kB
          Steve Loughran
        8. HADOOP-3733-branch-2-005.patch
          43 kB
          Steve Loughran
        9. HADOOP-3733-branch-2-006.patch
          43 kB
          Steve Loughran
        10. HADOOP-3733-branch-2-007.patch
          44 kB
          Steve Loughran

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HADOOP-14439 regression: secret stripping from S3x URIs breaks some downstream code

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13287 TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14114 S3A can no longer handle unencoded + in URIs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-20799 Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-10511 s3n:// incorrectly handles URLs with secret keys that contain a slash

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14022 S3 connections fails if in-URI AWS secret key contains '+' and '/' both

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-14833 Remove s3a user:secret authentication

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-13 filenames with ':' colon throws java.lang.IllegalArgumentException

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stuartsierra Stuart Sierra
              Votes:
              9 Vote for this issue
              Watchers:
              38 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: