Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16207

Improved S3A MR tests

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • fs/s3, test
    • None

    Description

      Reported failure of ITestDirectoryCommitMRJob in validation runs of HADOOP-16186; assertIsDirectory with s3guard enabled and a parallel test run: Path "is recorded as deleted by S3Guard"

          waitForConsistency();
    assertIsDirectory(outputPath) /* here */

      The file is there but there's a tombstone. Possibilities

      • some race condition with another test
      • tombstones aren't timing out
      • committers aren't creating that base dir in a way which cleans up S3Guard's tombstones.

      Remember: we do have to delete that dest dir before the committer runs unless overwrite==true, so at the start of the run there will be a tombstone. It should be overwritten by a success.

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue HADOOP-14936 S3Guard: remove "experimental" from documentation

          • Major - Major loss of function.
          • Resolved
          depends upon

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6521 MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Sub-task - The sub-task of the issue HADOOP-16184 S3Guard: Handle OOB deletions and creation of a file which has a tombstone marker

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue HADOOP-16422 AbstractITCommitMRJob.testMRJob test failures

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-16415 Speed up S3A test runs

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HADOOP-16186 S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1115

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: