Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33019

Use spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=1 by default

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.0.0, 3.0.1, 3.1.0
    • 3.0.2, 3.1.0
    • Spark Core

    Description

      By default, Spark should use a safe file output committer algorithm to avoid MAPREDUCE-7282.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7282 MR v2 commit algorithm should be deprecated and not the default

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-7300 PathOutputCommitter to add method failedTaskAttemptCommitRecoverable()

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #29895 [Github] Pull Request #29895 (dongjoon-hyun)

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: