Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-607 More efficient cube building
  3. KYLIN-781

Star Cubing alike partial cuboid flow down

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Later
    • None
    • None
    • Job Engine
    • None
    • Sprint 42, Sprint 43, Sprint 44, Sprint 45, Sprint 46

    Description

      Key idea

      • Assume base cuboid fit in mem (for a mapper split of course)
      • For child cuboid having common dim prefix, flow down records to child as soon as completed records can be fulfilled in child
      • Keep completed records in mem as long as possible, but also save them to disk in the background
      • When mem run short, pause flow down, remove completed records from mem, and resume flow from disk
      • Multiple flow downs at the same time, configurable thread number

      Attachments

        Activity

          People

            liyang.gmt8@gmail.com liyang
            liyang.gmt8@gmail.com liyang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: