Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19101

Apply rule [HiveJoinPushTransitivePredicatesRule] is getting stuck when there are huge number of predicates

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2
    • None
    • CBO
    • None

    Description

      Hive query is getting stuck during the optimisation phase while applying HiveJoinPushTransitivePredicatesRule when there are huge number of predicates.

       

      DEBUG Log:

      2018-04-04T11:22:47,991 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10963: Apply rule [ReduceExpressionsRule(Join)] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
      2018-04-04T11:22:48,359 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10964: Apply rule [HiveJoinAddNotNullRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]
      2018-04-04T11:22:48,360 [user: ganeshas] -1 DEBUG [6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main] plan.RelOptPlanner: call#10965: Apply rule [HiveJoinPushTransitivePredicatesRule] to [rel#881:HiveJoin.HIVE.[](left=HepRelVertex#879,right=HepRelVertex#887,condition==($1, $73),joinType=inner,algorithm=none,cost=not available)]

       

      Thread Status:

       

      "6f5e8faa-505c-48e3-a2cd-ce7bfced27f0 main" prio=5 tid=0x00007ff18e006800 nid=0x1c03 runnable [0x0000700008176000]
       java.lang.Thread.State: RUNNABLE
       at java.util.Arrays.copyOfRange(Arrays.java:2694)
       at java.lang.String.<init>(String.java:203)
       at java.lang.StringBuilder.toString(StringBuilder.java:405)
       at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:95)
       at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
       at org.apache.calcite.rex.RexCall.computeDigest(RexCall.java:84)
       at org.apache.calcite.rex.RexCall.toString(RexCall.java:100)
       at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.infer(HiveRelMdPredicates.java:516)
       at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference.inferPredicates(HiveRelMdPredicates.java:426)
       at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates.getPredicates(HiveRelMdPredicates.java:186)
       at GeneratedMetadataHandler_Predicates.getPredicates_$(Unknown Source)
       at GeneratedMetadataHandler_Predicates.getPredicates(Unknown Source)
       at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:721)
       at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinPushTransitivePredicatesRule.onMatch(HiveJoinPushTransitivePredicatesRule.java:83)
       at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:314)
       at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:502)
       at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:381)
       at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:275)
       at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72)
       at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:206)
       at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:193)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1575)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1448)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1174)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1096)
       at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
       at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
       at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
       at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:905)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:920)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:330)
       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11206)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:251)
       at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:265)
       at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:527)
       at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1358)
       at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1499)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1268)
       at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
       at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:213)
       at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:447)
       at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:924)
       at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:848)
       at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:735)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:606)
       at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      Locked ownable synchronizers:
       - <0x00000007d5cfcab0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      

       

       

      This issue seems to be  because of optimisation done on CBO (Specifically on calcite transitive predicate inference rule) in hive 2.1.1 and later versions. 

      This issue is not there in hive1.2.1 version.

       

      The queries to reproduce this issue is in this attachment : queries

      There is a view (test_view) created by combining the results from 16 tables using UNION ALL. 

      The query is getting stuck when I run a select query which performs join between view (test_view) and a table (test1). 

       

      Attachments

        1. queries
          204 kB
          Ganesha Shreedhara

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ganeshas Ganesha Shreedhara
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: