Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13786

Add S3A committers for zero-rename commits to S3 endpoints

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-beta1
    • 3.1.0
    • fs/s3
    • None

    Description

      A goal of this code is "support O(1) commits to S3 repositories in the presence of failures". Implement it, including whatever is needed to demonstrate the correctness of the algorithm. (that is, assuming that s3guard provides a consistent view of the presence/absence of blobs, show that we can commit directly).

      I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie. not visible until the close()), if we need to use that to allow us to abort commit operations.

      Attachments

        1. cloud-intergration-test-failure.log
          334 kB
          Ewan Higgs
        2. HADOOP-13786-036.patch
          778 kB
          Steve Loughran
        3. HADOOP-13786-037.patch
          851 kB
          Steve Loughran
        4. HADOOP-13786-038.patch
          854 kB
          Steve Loughran
        5. HADOOP-13786-039.patch
          929 kB
          Steve Loughran
        6. HADOOP-13786-HADOOP-13345-001.patch
          126 kB
          Steve Loughran
        7. HADOOP-13786-HADOOP-13345-002.patch
          150 kB
          Steve Loughran
        8. HADOOP-13786-HADOOP-13345-003.patch
          170 kB
          Steve Loughran
        9. HADOOP-13786-HADOOP-13345-004.patch
          222 kB
          Steve Loughran
        10. HADOOP-13786-HADOOP-13345-005.patch
          225 kB
          Steve Loughran
        11. HADOOP-13786-HADOOP-13345-006.patch
          231 kB
          Steve Loughran
        12. HADOOP-13786-HADOOP-13345-006.patch
          231 kB
          Steve Loughran
        13. HADOOP-13786-HADOOP-13345-007.patch
          246 kB
          Steve Loughran
        14. HADOOP-13786-HADOOP-13345-009.patch
          454 kB
          Steve Loughran
        15. HADOOP-13786-HADOOP-13345-010.patch
          471 kB
          Steve Loughran
        16. HADOOP-13786-HADOOP-13345-011.patch
          471 kB
          Steve Loughran
        17. HADOOP-13786-HADOOP-13345-012.patch
          484 kB
          Steve Loughran
        18. HADOOP-13786-HADOOP-13345-013.patch
          487 kB
          Steve Loughran
        19. HADOOP-13786-HADOOP-13345-015.patch
          499 kB
          Steve Loughran
        20. HADOOP-13786-HADOOP-13345-016.patch
          501 kB
          Steve Loughran
        21. HADOOP-13786-HADOOP-13345-017.patch
          505 kB
          Steve Loughran
        22. HADOOP-13786-HADOOP-13345-018.patch
          512 kB
          Steve Loughran
        23. HADOOP-13786-HADOOP-13345-019.patch
          521 kB
          Steve Loughran
        24. HADOOP-13786-HADOOP-13345-020.patch
          549 kB
          Steve Loughran
        25. HADOOP-13786-HADOOP-13345-021.patch
          558 kB
          Steve Loughran
        26. HADOOP-13786-HADOOP-13345-022.patch
          564 kB
          Steve Loughran
        27. HADOOP-13786-HADOOP-13345-023.patch
          597 kB
          Steve Loughran
        28. HADOOP-13786-HADOOP-13345-024.patch
          604 kB
          Steve Loughran
        29. HADOOP-13786-HADOOP-13345-025.patch
          624 kB
          Steve Loughran
        30. HADOOP-13786-HADOOP-13345-026.patch
          656 kB
          Steve Loughran
        31. HADOOP-13786-HADOOP-13345-027.patch
          653 kB
          Steve Loughran
        32. HADOOP-13786-HADOOP-13345-028.patch
          669 kB
          Steve Loughran
        33. HADOOP-13786-HADOOP-13345-028.patch
          669 kB
          Steve Loughran
        34. HADOOP-13786-HADOOP-13345-029.patch
          674 kB
          Steve Loughran
        35. HADOOP-13786-HADOOP-13345-030.patch
          690 kB
          Steve Loughran
        36. HADOOP-13786-HADOOP-13345-031.patch
          704 kB
          Steve Loughran
        37. HADOOP-13786-HADOOP-13345-032.patch
          733 kB
          Steve Loughran
        38. HADOOP-13786-HADOOP-13345-033.patch
          748 kB
          Steve Loughran
        39. HADOOP-13786-HADOOP-13345-035.patch
          771 kB
          Steve Loughran
        40. MAPREDUCE-6823-003.patch
          44 kB
          Steve Loughran
        41. objectstore.pdf
          187 kB
          Steve Loughran
        42. s3committer-master.zip
          42 kB
          Ryan Blue

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HADOOP-15631 Remove transient dependency on hadoop-hdfs-client

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7014 Fix java doc errors in jdk1.8

          • Major - Major loss of function.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-14714 handle InternalError in bulk object delete through retries

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14717 Add StreamCapabilities support to s3a

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14423 s3guard will set file length to -1 on a putObjectDirect(stream, -1) call

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15228 S3A Retry policy to retry on NoResponseException

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6961 Pull up FileOutputCommitter.getOutputPath to PathOutputCommitter

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          depends upon

          Sub-task - The sub-task of the issue HADOOP-13449 S3Guard: Implement DynamoDBMetadataStore.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          duplicates

          Sub-task - The sub-task of the issue HADOOP-13205 S3A to support custom retry policies; failfast on unknown host

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13811 s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14303 Review retry logic on all S3 SDK calls, implement where needed

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14381 S3AUtils.translateException to map 503 reponse to => throttling failure

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory

          • Major - Major loss of function.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-13205 S3A to support custom retry policies; failfast on unknown host

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13811 s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13967 S3ABlockOutputStream to support plugin point for different multipart strategies

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13968 S3a FS to support "__magic" path for the special "unmaterialized" writes

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13969 S3A to support commit(path) operation, which commits all pending put commits in a path

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14303 Review retry logic on all S3 SDK calls, implement where needed

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14381 S3AUtils.translateException to map 503 reponse to => throttling failure

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14859 Shaded AWS library stops s3a recognising ConnectTimeoutException

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-13761 S3Guard: implement retries for DDB failures and throttling; translate exceptions

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14831 Über-jira: S3a phase IV: Hadoop 3.1 features

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue HADOOP-14971 Merge S3A committers into trunk

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15003 Merge S3A committers into trunk: Yetus patch checker

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13574 Unnecessary file existence check causes problems with S3

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-13912 S3a Multipart Committer (avoid rename)

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-15087 S3A to support writing directly to the destination dir without creating temp directory to avoid rename

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HDFS-13713 Add specification of Multipart Upload API to FS specification, with contract tests

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15079 ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14303 Review retry logic on all S3 SDK calls, implement where needed

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14584 WASB to support high-performance commit protocol

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15890 Some S3A committer tests don't match ITest* pattern; don't run in maven

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HBASE-20431 Store commit transaction for filesystems that do not support an atomic rename

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SPARK-10063 Remove DirectParquetOutputCommitter

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-18883 FileNotFoundException on _temporary directory

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14161 Failed to rename file in S3A during FileOutputFormat commitTask

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-13912 S3a Multipart Committer (avoid rename)

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-18600 Hadoop 2.x should support s3a committers

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6974 Add standard configuration keys for HTrace values, propagate across to MR committers if set

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-7060 Cherry Pick PathOutputCommitter class/factory to branch-3.0

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-13846 S3A to implement rename(final Path src, final Path dst, final Rename... options)

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HIVE-16295 Add support for using Hadoop's S3A OutputCommitter

          • Major - Major loss of function.
          • Patch Available

          Bug - A problem which impairs or prevents the functions of the product. SPARK-18512 FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-22217 ParquetFileFormat to support arbitrary OutputCommitters

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14161 Failed to rename file in S3A during FileOutputFormat commitTask

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              11 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: