Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3171

Bucketed sort merge join doesn't work when multiple files exist for small alias

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Executing a query with the MAPJOIN hint and the bucketed sort merge join optimizations enabled:

      set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
set hive.optimize.bucketmapjoin = true;
set hive.optimize.bucketmapjoin.sortedmerge = true;

      works fine with partitioned tables if there is only one partition in the table. However, if you add a second partition, Hive attempts to do a regular map-side join which can fail because the tables are too large. Hive ought to be able to still do the bucketed sort merge join with partitions.

      Attachments

        1. HIVE-3171.2.patch.txt
          88 kB
          Navis Ryu
        2. HIVE-3171.1.patch.txt
          82 kB
          Navis Ryu

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-3290 BucketizedHiveInputFormat should support combining files having same bucket number

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-3218 Stream table of SMBJoin/BucketMapJoin with two or more partitions is not handled properly

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-3210 Support Bucketed mapjoin on partitioned table which has two or more partitions

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              navis Navis Ryu
              fwiffo Joey Echeverria
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: