Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16844

FS/FC createFile() builder to add a lazy-check option for async probes of the store

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.3.0
    • None
    • fs, fs/azure, fs/s3
    • None

    Description

      For object stores, file creation can triggers checks for: path being to a file, path not being a directory, parent path existing.

      These are all done before an output stream is returned -even though nothing may be manifest in the remote store until the screen is flashed or closed.

      If we supported lazy checks we could do those probes against the store while the application was actually generating data to write.

      Changing the behaviour of open() would be too fundamental. What we can do is add a new option to the createFile() builder which stores can support. When set, the caller is declaring they are happy with late reporting of failures -with any problems surfacing during a write/flush/close. It is notable with S3A permissions issues only surface later on anyway.

      Attachments

        Activity

          People

            Unassigned Unassigned
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: