XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.9.1, 2.8.4
    • 2.10.0, 3.2.0, 3.0.4, 3.1.2
    • fs
    • None

    Description

      FileSystem Globber does a listStatus(path) and then, if only one element is returned, getFileStatus(path).isDirectory() to see if it is a dir. The way getFileStatus() is wrapped, IOEs are downgraded to null

      On S3, if the path has had entries deleted, the listing may include files which are no longer there, so the getFileStatus(path),isDirectory triggers an NPE

      While its wrong to glob against S3 when its being inconsistent, we should at least fail gracefully here.

      Proposed

      1. log all IOEs raised in Globber.getFileStatus @ debug
      2. catch FNFEs and downgrade to warn
      3. continue

      The alternative would be fail fast on FNFE, but that's more traumatic

      Attachments

        1. HADOOP-15748-001.patch
          1 kB
          Steve Loughran

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: