Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-28207

NullPointerException is thrown when checking column uniqueness

    XMLWordPrintableJSON

Details

    Description

      In some cases, we skip checking null. For example, the last statement in the following set of queries fails with NPE.

      CREATE TABLE `store_sales` (`ss_item_sk` bigint);
      CREATE TABLE `household_demographics` (`hd_demo_sk` bigint);
      CREATE TABLE `item` (`i_item_sk` bigint);
      ALTER TABLE `store_sales` ADD CONSTRAINT `pk_ss` PRIMARY KEY (`ss_item_sk`) DISABLE NOVALIDATE RELY;
      ALTER TABLE `item` ADD CONSTRAINT `pk_i` PRIMARY KEY (`i_item_sk`) DISABLE NOVALIDATE RELY;
      ALTER TABLE `store_sales` ADD CONSTRAINT `ss_i` FOREIGN KEY (`ss_item_sk`) REFERENCES `item`(`i_item_sk`) DISABLE NOVALIDATE RELY;
      
      EXPLAIN
      SELECT i_item_sk
      FROM store_sales, household_demographics, item
      WHERE ss_item_sk = i_item_sk

      The NPE happens with HiveJoinConstraintsRule in the above case.

      org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: NullPointerException null
           at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:214) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.operation.Operation.run(Operation.java:286) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:557) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:542) ~[hive-service-4.0.0.jar:4.0.0]
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
           at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
           at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-4.0.0.jar:4.0.0]
           at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_275]
           at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_275]
           at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.3.6.jar:?]
           at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-4.0.0.jar:4.0.0]
           at com.sun.proxy.$Proxy42.executeStatementAsync(Unknown Source) ~[?:?]
           at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:316) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:652) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[hive-exec-4.0.0.jar:4.0.0]
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275]
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275]
           at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
      Caused by: java.lang.NullPointerException
           at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule.onMatch(HiveJoinConstraintsRule.java:277) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2446) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2405) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1943) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1686) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1567) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1319) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:570) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13149) ~[hive-exec-4.0.0.jar:4.0.0] 

      Attachments

        Issue Links

          Activity

            People

              okumin Shohei Okumiya
              okumin Shohei Okumiya
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: