Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Later
-
None
-
None
-
None
-
Sprint 42, Sprint 43, Sprint 44, Sprint 45, Sprint 46
Description
Key idea
- Assume base cuboid fit in mem (for a mapper split of course)
- For child cuboid having common dim prefix, flow down records to child as soon as completed records can be fulfilled in child
- Keep completed records in mem as long as possible, but also save them to disk in the background
- When mem run short, pause flow down, remove completed records from mem, and resume flow from disk
- Multiple flow downs at the same time, configurable thread number