Description
Root Cause
The Spark org.apache.spark.sql.functions.expr function cannot recognize special characters such as Chinese characters and needs to be enclosed in backticks.
Dev Design
Identify the places where expr is called and add compatibility handling for special characters:
Build the computable column
FlatTableAndDictBase#concatCCs
Build dictionary
DFBuilderHelper#selectColumnsInTable, DFBuilderHelper#chooseSuitableCols
Build dictionary
DFDictionaryBuilder#wrapCol
Root Cause
Spark org.apache.spark.sql.functions.expr 函数不能识别中文等特殊字符,需要用反引号包裹。
Dev Design
找出 expr 调用处,增加对特殊字符的兼容处理:
- 构建可计算列 FlatTableAndDictBase#concatCCs
- 构建字典 DFBuilderHelper#selectColumnsInTable、DFBuilderHelper#chooseSuitableCols
- 构建字典 DFDictionaryBuilder#wrapCol
{{}}