Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14323

Reduce number of FS permissions and redundant FS operations

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • None
    • None

    Description

      Some examples are given below.
      1. When creating stage directory, FileUtils sets the directory permissions by running a set of chgrp and chmod commands. In systems like S3, this would not be relevant.
      2. In some cases, fs.delete() is followed by fs.exists(). In this case, it might be redundant to check for exists() (lookup ops are expensive in systems like S3).

      Attachments

        1. HIVE-14323.1.patch
          10 kB
          Rajesh Balamohan
        2. HIVE-14323.2.patch
          11 kB
          Rajesh Balamohan
        3. HIVE-14323.3.patch
          11 kB
          Rajesh Balamohan

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. HIVE-18702 INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14269 Performance optimizations for data on S3

          • Major - Major loss of function.
          • Open
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13427 Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-16736 remove redundant FileSystem status checks calls from Spark codebase

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-13925 ETL optimizations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: