Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1623

Transform Reshuffle directly in Spark runner

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 0.6.0
    • runner-spark
    • None

    Description

      Transform Reshuffle directly in Spark runner.
      Spark's repartition is logically equivalent to Reshuffle and will cause the required fusion break without incurring the overhead of groupByKey (In streaming pipelines this will use updateStateByKey) compared to a simple repartition.

      Attachments

        Issue Links

          Activity

            People

              aviemzur Aviem Zur
              aviemzur Aviem Zur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: