Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13427

Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.9.0, 3.0.0-alpha2
    • fs
    • None
    • Reviewed

    Description

      We're cleaning up Hive and Spark's use of FileSystem.exists, because it is often the case we see code of exists+open, exists+delete, when the exists probe is needless. Against object stores, expensive needless.

      Hadoop can set an example here by stripping them out. It will also show where there are opportunities to optimise things better and/or improve reporting.

      Attachments

        1. HADOOP-13427.009.patch
          60 kB
          Mingliang Liu
        2. HADOOP-13427.009.patch
          60 kB
          Mingliang Liu
        3. HADOOP-13427.008.patch
          60 kB
          Mingliang Liu
        4. HADOOP-13427.008.patch
          60 kB
          Mingliang Liu
        5. HADOOP-13427.007.patch
          60 kB
          Steve Loughran
        6. HADOOP-13427.006.patch
          60 kB
          Mingliang Liu
        7. HADOOP-13427.005.patch
          60 kB
          Mingliang Liu
        8. HADOOP-13427.004.patch
          60 kB
          Mingliang Liu
        9. HADOOP-13427.003.patch
          56 kB
          Mingliang Liu
        10. HADOOP-13427-002.patch
          53 kB
          Steve Loughran
        11. HADOOP-13427-001.patch
          53 kB
          Steve Loughran

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17255 JavaKeyStoreProvider fails to create a new key if the keystore is HDFS

          • Major - Major loss of function.
          • Reopened
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-3540 Eliminate inefficient use of FileSystem API

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-16736 remove redundant FileSystem status checks calls from Spark codebase

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14323 Reduce number of FS permissions and redundant FS operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              liuml07 Mingliang Liu
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: