Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15229

Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.3.0
    • fs, fs/azure, fs/s3
    • None

    Description

      Replicate HDFS-1170 and HADOOP-14365 with an API to open files.

      A key requirement of this is not HDFS, it's to put in the fadvise policy for working with object stores, where getting the decision to do a full GET and TCP abort on seek vs smaller GETs is fundamentally different: the wrong option can cost you minutes. S3A and Azure both have adaptive policies now (first backward seek), but they still don't do it that well.

      Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise" "random" as an option when they open files; I can imagine other options too.

      The Builder model of eddyxu is the one to mimic, method for method. Ideally with as much code reuse as possible

      Attachments

        1. HADOOP-15229-020.patch
          394 kB
          Steve Loughran
        2. HADOOP-15229-019.patch
          395 kB
          Steve Loughran
        3. HADOOP-15229-018.patch
          393 kB
          Steve Loughran
        4. HADOOP-15229-017.patch
          394 kB
          Steve Loughran
        5. HADOOP-15229-016.patch
          389 kB
          Steve Loughran
        6. HADOOP-15229-015.patch
          367 kB
          Steve Loughran
        7. HADOOP-15229-014.patch
          357 kB
          Steve Loughran
        8. HADOOP-15229-013.patch
          355 kB
          Steve Loughran
        9. HADOOP-15229-012.patch
          324 kB
          Steve Loughran
        10. HADOOP-15229-011.patch
          289 kB
          Steve Loughran
        11. HADOOP-15229-010.patch
          267 kB
          Steve Loughran
        12. HADOOP-15229-009.patch
          250 kB
          Steve Loughran
        13. HADOOP-15229-007.patch
          250 kB
          Steve Loughran
        14. HADOOP-15229-006.patch
          237 kB
          Steve Loughran
        15. HADOOP-15229-005.patch
          229 kB
          Steve Loughran
        16. HADOOP-15229-004.patch
          194 kB
          Steve Loughran
        17. HADOOP-15229-004.patch
          197 kB
          Steve Loughran
        18. HADOOP-15229-003.patch
          151 kB
          Steve Loughran
        19. HADOOP-15229-002.patch
          27 kB
          Steve Loughran
        20. HADOOP-15229-001.patch
          23 kB
          Steve Loughran

        Issue Links

          causes

          Sub-task - The sub-task of the issue HADOOP-16480 S3 Select Exceptions are not being converted to IOEs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7184 TestJobCounters byte counters omitting crc file bytes read

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16106 hadoop-aws project javadoc does not compile

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-15364 Add support for S3 Select to S3A

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-15963 Add ABFS support for Async Scatter/Gather IO

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HADOOP-15364 Add support for S3 Select to S3A

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15964 Add S3A support for Async Scatter/Gather IO

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-7182 MapReduce input format/record readers to support S3 select queries

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SPARK-48571 Reduce the number of accesses to S3 object storage

          • Major - Major loss of function.
          • Open
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-19199 Include FileStatus when opening a file from FileSystem

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-19200 Reduce the number of headObject when opening a file with the s3 file system

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-11867 Add a high-performance vectored read API.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-15949 open(PathHandle) spec and implementations ambiguous about what to raise when path is deleted

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HDFS-14111 hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-18287 Provide a shim library for modern FS APIs

          • Major - Major loss of function.
          • Open
          is required by

          Sub-task - The sub-task of the issue HADOOP-15625 S3A input stream to use etags/version number to detect changed source files

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-13327 Add OutputStream + Syncable to the Filesystem Specification

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-14365 Stabilise FileSystem builder-based create API

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-15691 Add PathCapabilities to FS and FC to complement StreamCapabilities

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-2744 Extend FSDataInputStream to allow fadvise

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-11170 Add builder-based create API to FileSystem

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14478 Add libhdfs APIs for openFile

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: