[HADOOP-14439] regression: secret stripping from S3x URIs breaks some downstream code - ASF JIRA

XML

Word

Printable

JSON

Target Version/s:

2.8.3
Release Note:

Hide
We have restored the retention of a username:secret in an s3a: URL if you authenticate with S3 using embedded URI secrets. This is because some applications using strings to marshall the URLs were breaking. However, using secrets in this way is dangerous as it will end up in logs. It will be unsupported in Hadoop 3. To use different credentials in different buckets, move to per-bucket configuration

Show
We have restored the retention of a username:secret in an s3a: URL if you authenticate with S3 using embedded URI secrets. This is because some applications using strings to marshall the URLs were breaking. However, using secrets in this way is dangerous as it will end up in logs. It will be unsupported in Hadoop 3. To use different credentials in different buckets, move to per-bucket configuration

Spark is listing the contents of a path with getFileStatus(path), then looking up the path value doing a lookup of the contents.

Apparently the lookup is failing to find files if you have a secret in the key, s3a://key:secret@bucket/path.

Presumably this is because the stripped values aren't matching.

breaks

SPARK-20799 Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

is broken by

HADOOP-3733 "s3:" URLs break when Secret Key contains a slash, even if encoded

is part of

HADOOP-13204 Über-jira: S3a phase III: scale and tuning

relates to

HADOOP-14833 Remove s3a user:secret authentication