Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.15.0
-
None
-
None
Description
A SelectionVectorRemover is often used after a filter, to copy into a newly allocated new batch only the "filtered out" rows. In some cases the columns used by the filter are not needed downstream; currently these columns are being needlessly allocated and copied, and later removed by a Project.
Suggested improvement: The planner can pass the information about these columns to the SelectionVectorRemover, which would avoid this useless allocation and copy. The Planner would also eliminate that Project from the plan.
Here is an example, the query:
select max(l_quantity) from cp.`tpch/lineitem.parquet` L where L.l_orderkey > 58999 and L.l_shipmode = 'TRUCK' group by l_linenumber ;
And the result plan (trimmed for readability), where "l_orderkey" and "l_shipmode" are removed by the Project:
00-00 Screen : rowType = RecordType(ANY EXPR$0):
00-01 Project(EXPR$0=[$0]) : rowType = RecordType(ANY EXPR$0):
00-02 Project(EXPR$0=[$1]) : rowType = RecordType(ANY EXPR$0):
00-03 HashAgg(group=[\{0}], EXPR$0=[MAX($1)]) : rowType = RecordType(ANY l_linenumber, ANY EXPR$0):
00-04 Project(l_linenumber=[$2], l_quantity=[$3]) : rowType = RecordType(ANY l_linenumber, ANY l_quantity):
00-05 SelectionVectorRemover : rowType = RecordType(ANY l_orderkey, ANY l_shipmode, ANY l_linenumber, ANY l_quantity):
00-06 Filter(condition=[AND(>($0, 58999), =($1, 'TRUCK'))]) : rowType = RecordType(ANY l_orderkey, ANY l_shipmode, ANY l_linenumber, ANY l_quantity):
00-07 Scan(table=[[cp, tpch/lineitem.parquet]], groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=classpath:/tpch/lineitem.parquet]], selectionRoot=classpath:/tpch/lineitem.parquet, numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`l_orderkey`, `l_shipmode`, `l_linenumber`, `l_quantity`]]]) : rowType = RecordType(ANY l_orderkey, ANY l_shipmode, ANY l_linenumber, ANY l_quantity):
The implementation will not be simple, as the relevant code (e.g., GenericSV2Copier) has no idea of specific columns.