Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
  3. HADOOP-15503

strip s3.amazonaws.com off hostnames before making s3a calls

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.1.0
    • None
    • fs/s3
    • None

    Description

      If you copy an http URL https://bucketname.s3.amazonaws.com/ and convert to an s3a one by replacing the schema, you can try to list

      bin/hadoop fs -ls -r s3a://hwdev-steve-new.s3.amazonaws.com/
      

      But do that and you get told there's no such bucket

      ls: Bucket hwdev-steve-new.s3.amazonaws.com does not exist
      

      This is non-intuitive, and catches me out.

      We could strip this automatically during initialization to produce the actual bucket, which would need to be done before any per-bucket init is done.

      I do worry about what could break though

      Attachments

        Activity

          People

            Unassigned Unassigned
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: