Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-13914

s3guard: improve S3AFileStatus#isEmptyDirectory handling

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • HADOOP-13345
    • HADOOP-13345
    • fs/s3
    • None
    • Reviewed

    Description

      As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag stored in S3AFileStatus is missing from DynamoDBMetadataStore.

      The approach taken by LocalMetadataStore is not suitable for the DynamoDB implementation, and also sacrifices good code separation to minimize S3AFileSystem changes pre-merge to trunk.

      I will attach a design doc that attempts to clearly explain the problem and preferred solution. I suggest we do this work after merging the HADOOP-13345 branch to trunk, but am open to suggestions.

      I can also attach a patch of a integration test that exercises the missing case and demonstrates a failure with DynamoDBMetadataStore.

      Attachments

        1. test-only-HADOOP-13914.patch
          5 kB
          Aaron Fabbri
        2. s3guard-empty-dirs.md
          8 kB
          Aaron Fabbri
        3. HADOOP-13914-HADOOP-13345.008.patch
          69 kB
          Aaron Fabbri
        4. HADOOP-13914-HADOOP-13345.007.patch
          69 kB
          Aaron Fabbri
        5. HADOOP-13914-HADOOP-13345.006.patch
          69 kB
          Aaron Fabbri
        6. HADOOP-13914-HADOOP-13345.005.patch
          68 kB
          Aaron Fabbri
        7. HADOOP-13914-HADOOP-13345.004.patch
          67 kB
          Aaron Fabbri
        8. HADOOP-13914-HADOOP-13345.003.patch
          63 kB
          Aaron Fabbri
        9. HADOOP-13914-HADOOP-13345.002.patch
          62 kB
          Aaron Fabbri
        10. HADOOP-13914-HADOOP-13345.000.patch
          14 kB
          Mingliang Liu

        Issue Links

          Activity

            People

              fabbri Aaron Fabbri
              fabbri Aaron Fabbri
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: