Details
-
Improvement
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Transform Reshuffle directly in Spark runner.
Spark's repartition is logically equivalent to Reshuffle and will cause the required fusion break without incurring the overhead of groupByKey (In streaming pipelines this will use updateStateByKey) compared to a simple repartition.
Attachments
Issue Links
- links to