Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12810

FileSystem#listLocatedStatus causes unnecessary RPC calls

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.7.2
    • 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • fs, fs/s3
    • None
    • Reviewed

    Description

      FileSystem#listLocatedStatus lists the files in a directory and then calls getFileBlockLocations(stat.getPath(), ...) for each instead of getFileBlockLocations(stat, ...). That function with the path arg just calls getFileStatus to get another file status from the path and calls the file status version, so this ends up calling getFileStatus unnecessarily.

      This is particularly bad for S3, where getFileStatus is expensive. Avoiding the extra call improved input split calculation time for a data set in S3 by ~20x: from 10 minutes to 25 seconds.

      Attachments

        1. HADOOP-12810.1.patch
          2 kB
          Ryan Blue

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6637 Testcase Failure : TestFileInputFormat.testSplitLocationInfo

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-9926 Parallelize file listing for partitioned Hive table

          • Major - Major loss of function.
          • Resolved
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-10340 Use S3 bulk listing for S3-backed Hive tables

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              rdblue Ryan Blue
              rdblue Ryan Blue
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: