Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-13936

S3Guard: DynamoDB can go out of sync with S3AFileSystem.delete()

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Duplicate
    • 3.0.0-beta1, 3.1.0, 3.1.1
    • None
    • fs/s3
    • None

    Description

      As a part of S3AFileSystem.delete operation innerDelete is invoked, which deletes keys from S3 in batches (default is 1000). But DynamoDB is updated only at the end of this operation. This can cause issues when deleting large number of keys.

      E.g, it is possible to get exception after deleting 1000 keys and in such cases dynamoDB would not be updated. This can cause DynamoDB to go out of sync.

      Attachments

        Issue Links

          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-14936 S3Guard: remove "experimental" from documentation

          • Major - Major loss of function.
          • Resolved
          is part of

          Sub-task - The sub-task of the issue HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-15191 Add Private/Unstable BulkDelete operations to supporting object stores for DistCP

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15193 add bulk delete call to metastore API & DDB impl

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-13980 S3Guard CLI: Add fsck check and fix commands

          • Major - Major loss of function.
          • In Progress

          Activity

            People

              stevel@apache.org Steve Loughran
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: