Details
-
Bug
-
Status: Open
-
Blocker
-
Resolution: Unresolved
-
2.3.4, 3.0.0
-
None
-
None
Description
drop table temp.test_cbo_right; CREATE EXTERNAL TABLE temp.test_cbo_right( `rid` string, `dtm` string, `map` map<string, string>, `tags` array <string> );drop table temp.test_cbo_left; CREATE EXTERNAL TABLE temp.test_cbo_left( `lid` string, `dtm` string ) explain select b.tags from temp.test_cbo_left a left join temp.test_cbo_right b on a.lid = b.rid where b.dtm = '20200814' and a.dtm = '20200813'
The Problem will occur in left join and select array type or map type column in right table(reverse for right join or both for full join).
code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from calcite and then convert it back to `optimizedAST`, the projection of b.tags has been optimized to be RexCall => CAST(b.tags) in calcite code. Normally the projection should be an RexInputRef, but in this case, calcite need cast `not null` to `nullable`. However, TypeConverter in ASTConverter has not support ARRAY and MAP.
NPE threw here:
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor visitCall
HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text); if (ht.args != null) { for (String castArg : ht.args) astBldr.add(HiveParser.Identifier, castArg); }
The final result is cbo failed and skip CalcitePlan and cause partition pruning failed.
when table is large, sql will explode.
I wonder which project should fix this problem