Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features
  3. HADOOP-13525

Optimize uses of FS operations in the ASF analysis frameworks and libraries

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Done
    • 2.8.1
    • 3.3.1
    • fs, fs/s3
    • None

    Description

      Review uses of the FS APIs in applications using the Hadoop FS API to access filesystems; identify suboptimal uses and tune them for better performance against HDFS and object stores

      • Assume arbitrary Hadoop 2.x releases: make no changes which are known to make operations on older versions of Hadoop slower
      • Do propose those changes which deliver speedups in later versions of Hadoop, while not impacting older versions, or risk of causing scalability problems.
      • Add more tests, especially scalable ones which also display metrics.
      • Use standard benchmarks and optimization tools to identify hotspots.
      • Use FS behaviour as verified in the FS contract tests as evidence that filesystems correctly implement the Hadoop FS APIs. If a use of an API call is made which hints at the expectation of different/untested behaviours, leave alone and add new tests to the Hadoop FS contract to determine cross-FS semantics.
      • Focus on the startup, split calculation and directory scanning operations: the ones which slow down entire queries.
      • Eliminate use of isDirectory(), getLength(), exists() if a followon operation (getStatus(),delete(), ... makes the use redundant.
      • Assume that FileStatus entries are not cached; the cost of creating them is 1 RPC call against HDFS, 1+ HTTPS call against object stores.
      • Locate calls to the listing operations, identify speedups, especially on recursive directory scans.
      • Identify suboptimal seek patterns (backwards as well as forwards) and attempt to reduce/eliminate through reordering and result caching.
      • Try to reuse the results of previous operations (e.g FileStatus instances) in follow-on calls.
      • Commonly used file formats (e.g ORC) will have transitive benefits.
      • Frameworks to use predicate pushdown where this delivers speedups
      • Document best practises identified and implemented.

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue SPARK-16980 Load only catalog table partition metadata required to answer a query

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HIVE-14165 Remove Hive file listing during split computation

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14269 Performance optimizations for data on S3

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6760 LocatedFileStatusFetcher to use listFiles(recursive)

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6800 FileInputFormat.singleThreadedListStatus to use listFiles(recursive)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-17861 Store data source partitions in metastore and push partition pruning into metastore

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13427 Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-14551 Reduce number of NameNode calls in OrcRelation with FileSourceStrategy mode

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-16736 remove redundant FileSystem status checks calls from Spark codebase

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-17159 Improve FileInputDStream.findNewFiles list performance

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-18917 Dataframe - Time Out Issues / Taking long time in append mode on object stores

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-17179 Consider improving partition pruning in HiveMetastoreCatalog

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14323 Reduce number of FS permissions and redundant FS operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14423 S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. PIG-4442 Eliminate redundant RPC call to get file information in HPath.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: