Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4860

Partition pruning does not work when we have a correlated sub-query

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.8.0
    • None
    • None

    Description

      git commit # : 0a4c21cc15329c063f256f6fbf2c4c69a90d9fa1

      The below query indicates that partition pruning did not occur for the sub-query

      explain plan for select count (*) from (
        select t1.l_orderkey, t1.l_linenumber from dfs.`/drill/testdata/partition_pruning/nested/l_3level` t1
        where
          t1.dir0 = 1 and t1.dir1='one' and t1.dir2 = '2015-7-12' and t1.l_orderkey in (
            select t2.l_orderkey from dfs.`/drill/testdata/partition_pruning/nested/l_3level` t2 where t2.dir0 = 1 and t2.dir1='two' and t2.dir2 = '2015-8-12' and t2.l_discount > t1.l_discount
          )
      ) data;
      
      +------+------+
      | text | json |
      +------+------+
      | 00-00    Screen
      00-01      Project(EXPR$0=[$0])
      00-02        StreamAgg(group=[{}], EXPR$0=[COUNT()])
      00-03          Project($f0=[0])
      00-04            Project(l_discount=[$2], l_orderkey=[$3], l_orderkey0=[$0], l_discount0=[$1])
      00-05              HashJoin(condition=[AND(=($2, $1), =($3, $0))], joinType=[inner])
      00-07                HashAgg(group=[{0, 1}])
      00-09                  Project(l_orderkey=[$0], l_discount0=[$2])
      00-10                    Project(l_orderkey=[$1], l_discount=[$2], l_discount0=[$0])
      00-11                      HashJoin(condition=[=($2, $0)], joinType=[inner])
      00-13                        HashAgg(group=[{0}])
      00-15                          Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/two/2015-8-12/30.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/three/2015-7-12/40.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-13/20.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-12/10.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-9-12/50.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-7-12/30.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/two/2015-8-12/40.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/three/2015-8-12/10.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-9-12/20.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-7-12/50.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/2/one/2015-8-12/10.parquet]], selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level, numFiles=11, usedMetadataFile=false, columns=[`l_discount`]]])
      00-12                        Project(l_orderkey=[$0], l_discount0=[$1])
      00-14                          Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/two/2015-8-12/30.parquet]], selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level, numFiles=1, usedMetadataFile=false, columns=[`l_orderkey`, `l_discount`]]])
      00-06                Project(l_discount=[$0], l_orderkey0=[$1])
      00-08                  Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/partition_pruning/nested/l_3level/1/one/2015-7-12/10.parquet]], selectionRoot=maprfs:/drill/testdata/partition_pruning/nested/l_3level, numFiles=1, usedMetadataFile=false, columns=[`l_discount`, `l_orderkey`]]])
      

      Attachments

        1. l_3level.tgz
          13 kB
          Rahul Kumar Challapalli

        Activity

          People

            Unassigned Unassigned
            rkins Rahul Kumar Challapalli
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: