Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13164

Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.8.0
    • 2.9.0, 3.0.0-alpha2
    • fs/s3
    • None

    Description

      https://github.com/apache/hadoop/blob/27c4e90efce04e1b1302f668b5eb22412e00d033/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1224

      deleteUnnecessaryFakeDirectories is invoked in S3AFileSystem during rename and on outputstream close() to purge any fake directories. Depending on the nesting in the folder structure, it might take a lot longer time as it invokes getFileStatus multiple times. Instead, it should be able to break out of the loop once a non-empty directory is encountered.

      Attachments

        1. HADOOP-13164-branch-2-004.patch
          11 kB
          Rajesh Balamohan
        2. HADOOP-13164-branch-2-003.patch
          11 kB
          Rajesh Balamohan
        3. HADOOP-13164-branch-005.patch
          13 kB
          Steve Loughran
        4. HADOOP-13164.branch-2-002.patch
          11 kB
          Rajesh Balamohan
        5. HADOOP-13164.branch-2.WIP.patch
          8 kB
          Rajesh Balamohan
        6. HADOOP-13164.branch-2.WIP.002.patch
          12 kB
          Rajesh Balamohan

        Issue Links

          causes

          Sub-task - The sub-task of the issue HADOOP-16090 S3A Client to add explicit support for versioned stores

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-13230 S3A to optionally retain directory markers

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14594 ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: