Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
  3. HADOOP-13968

S3a FS to support "__magic" path for the special "unmaterialized" writes

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.0.0-beta1
    • None
    • fs/s3
    • None

    Description

      S3AFileSystem to add support for a special path, such as .temp_pending_put/ or similar, which, when used as the base of a path, indicates that the file is actually to be saved to the parent dir, but only via a delayed put commit operation.

      At the same time, we may need blocks on some normal fileIO ops under these dirs, especially rename and delete, as this would cause serious problems including data loss and large bills for pending data.

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: