Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9277

Hybrid Hybrid Grace Hash Join

    XMLWordPrintableJSON

Details

    Description

      We are proposing an enhanced hash join algorithm called “hybrid hybrid grace hash join”.

      We can benefit from this feature as illustrated below:

      • The query will not fail even if the estimated memory requirement is slightly wrong
      • Expensive garbage collection overhead can be avoided when hash table grows
      • Join execution using a Map join operator even though the small table doesn't fit in memory as spilling some data from the build and probe sides will still be cheaper than having to shuffle the large fact table

      The design was based on Hadoop’s parallel processing capability and significant amount of memory available.

      Attachments

        1. HIVE-9277.15.patch
          138 kB
          Wei Zheng
        2. HIVE-9277.14.patch
          138 kB
          Wei Zheng
        3. HIVE-9277.13.patch
          138 kB
          Wei Zheng
        4. HIVE-9277.08.patch
          114 kB
          Wei Zheng
        5. HIVE-9277.07.patch
          88 kB
          Wei Zheng
        6. HIVE-9277.06.patch
          86 kB
          Wei Zheng
        7. HIVE-9277.05.patch
          87 kB
          Wei Zheng
        8. HIVE-9277.04.patch
          95 kB
          Wei Zheng
        9. HIVE-9277.03.patch
          84 kB
          Wei Zheng
        10. HIVE-9277.02.patch
          86 kB
          Wei Zheng
        11. HIVE-9277.01.patch
          89 kB
          Wei Zheng
        12. High-leveldesignforHybridHybridGraceHashJoinv1.0.pdf
          453 kB
          Wei Zheng

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HIVE-10287 Implement Hybrid Hybrid Grace Hash Join for Spark Branch [Spark Branch]

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HIVE-10123 Hybrid grace Hash join : Use estimate key count from stats to initialize BytesBytesMultiHashMap

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-10284 enable container reuse for grace hash join

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-9789 Hybrid Hybrid Grace Hash Join: improve hashtable serialization

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-9790 Hybrid Hybrid Grace Hash Join: improve side file serialization

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-10072 Add vectorization support for Hybrid Grace Hash Join

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-10403 Add n-way join support for Hybrid Grace Hash Join

          • Major - Major loss of function.
          • Closed
          is required by

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-11306 Add a bloom-1 filter for Hybrid MapJoin spills

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Review Board

          Activity

            People

              wzheng Wei Zheng
              wzheng Wei Zheng
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: