Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16058

S3A tests to include Terasort

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0, 3.2.1
    • fs/s3, test
    • None

    Description

      Add S3A tests to run terasort for the magic and directory committers.

      MAPREDUCE-7091 is a requirement for this

      Bonus feature: print the results to see which committers are faster in the specific test setup. As that's a function of latency to the store, bandwidth and size of jobs, it's not at all meaningful, just interesting.

      Attachments

        1. HADOOP-16058-002.patch
          55 kB
          Steve Loughran
        2. HADOOP-16058-002.patch
          55 kB
          Steve Loughran
        3. HADOOP-16058-002.patch
          55 kB
          Steve Loughran
        4. HADOOP-16058-001.patch
          32 kB
          Steve Loughran

        Issue Links

          causes

          Sub-task - The sub-task of the issue HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          contains

          Sub-task - The sub-task of the issue MAPREDUCE-7090 BigMapOutput example doesn't work with paths off cluster fs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-7091 Terasort on S3A to switch to new committers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-7092 MR examples to work better against cloud stores

          • Major - Major loss of function.
          • Resolved
          depends upon

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7099 Daily test result fails in MapReduce JobClient though there isn't any error

          • Critical - Crashes, loss of data, severe memory leak.
          • Reopened
          incorporates

          Sub-task - The sub-task of the issue HADOOP-15890 Some S3A committer tests don't match ITest* pattern; don't run in maven

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-7090 BigMapOutput example doesn't work with paths off cluster fs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-7091 Terasort on S3A to switch to new committers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-7092 MR examples to work better against cloud stores

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16153 Allow TeraGen to use schema-specific output committer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link GitHub Pull Request #530

          Web Link GitHub Pull Request #565

          Web Link GitHub Pull Request #576

          Web Link GitHub Pull Request #577

          Web Link GitHub Pull Request #631

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: