Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express
  3. HADOOP-18948

S3A. Add option fs.s3a.directory.operations.purge.uploads to purge on rename/delete

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      S3A directory delete and rename will optionally abort all pending uploads
      under the to-be-deleted paths when
      fs.s3a.directory.operations.purge.upload is true
      It is off by default.
      Show
      S3A directory delete and rename will optionally abort all pending uploads under the to-be-deleted paths when fs.s3a.directory.operations.purge.upload is true It is off by default.

    Description

      On third-party stores without lifecycle rules its possible to accrue many GB of pending multipart uploads, including from

      • magic committer jobs where spark driver/MR AM failed before commit/abort
      • distcp jobs which timeout and get aborted
      • any client code writing datasets which are interrupted before close.

      Although there's a purge pending uploads option, that's dangerous because if any fs is instantiated with it, it can destroy in-flight work

      otherwise, the "hadoop s3guard uploads" command does work but needs scheduling/manual execution

      proposed: add a new property fs.s3a.directory.operations.purge.uploads which will automatically cancel all pending uploads under a path

      • delete: everything under the dir
      • rename: all under the source dir

      This will be done in parallel to the normal operation, but no attempt to post abortMultipartUploads in different threads. The assumption here is that this is rare. And it'll be off by default as in AWS people should have rules for these things.

      + doc (third_party?)
      + add new counter/metric for abort operations, count and duration
      + test to include cost assertions

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: