Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.15.0
-
None
-
None
Description
Performing data bucketing execution: two tables (orders, orders_item), divided into buckets (bucketing) based on the same fields (orderid) and the same number of buckets. In join by order id, join and aggregation calculations can be performed independently, because the same order ids of both tables are divided into buckets with the same ids.
This has several advantages:
1. Whenever a bucket of data is computed, the memory occupied by this bucket can be released immediately, so memory consumption can be limited by controlling the number of buckets processed in parallel.
2. reduces a lot of shuffling