Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17400 Optimize S3A for maximum performance in directory listings
  3. HADOOP-16458

LocatedFileStatusFetcher scans failing intermittently against S3 store

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • fs/s3
    • None

    • S3 + S3Guard

    Description

      Intermittent failure of LocatedFileStatusFetcher.getFileStatuses(), which is using globStatus to find files.

      I'd say "turn s3guard on" except this appears to be the case, and the dataset being read is
      over 1h old.

      Which means it is harder than I'd like to blame S3 for what would sound like an inconsistency

      We're hampered by the number of debug level statements in the globber code being approximately none; there's no debugging to turn on. All we know is that globFiles returns null without any explanation.

      Attachments

        Issue Links

          breaks

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-24484 Upgrade Hadoop to 3.3.1 And Tez to 0.10.2

          • Major - Major loss of function.
          • Closed
          causes

          Sub-task - The sub-task of the issue HADOOP-16626 S3A ITestRestrictedReadAccess fails

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-17105 S3AFS globStatus attempts to resolve symlinks

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-13373 Add S3A implementation of FSMainOperationsBaseTest

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-16465 Tune S3AFileSystem.listLocatedStatus

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-15748 S3 listing inconsistency can raise NPE in globber

          • Major - Major loss of function.
          • Resolved
          supercedes

          Sub-task - The sub-task of the issue HADOOP-13371 S3A globber to use bulk listObject call over recursive directory scan

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1160

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: