Details
-
Sub-task
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.4.0
-
Reviewed
-
Description
On third-party stores without lifecycle rules its possible to accrue many GB of pending multipart uploads, including from
- magic committer jobs where spark driver/MR AM failed before commit/abort
- distcp jobs which timeout and get aborted
- any client code writing datasets which are interrupted before close.
Although there's a purge pending uploads option, that's dangerous because if any fs is instantiated with it, it can destroy in-flight work
otherwise, the "hadoop s3guard uploads" command does work but needs scheduling/manual execution
proposed: add a new property fs.s3a.directory.operations.purge.uploads which will automatically cancel all pending uploads under a path
- delete: everything under the dir
- rename: all under the source dir
This will be done in parallel to the normal operation, but no attempt to post abortMultipartUploads in different threads. The assumption here is that this is rare. And it'll be off by default as in AWS people should have rules for these things.
+ doc (third_party?)
+ add new counter/metric for abort operations, count and duration
+ test to include cost assertions
Attachments
Issue Links
- is depended upon by
-
SPARK-47008 Spark to support S3 Express One Zone Storage
- Open
- links to