Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6173

Support transitive closure during filter push down and partition pruning

    XMLWordPrintableJSON

Details

    Description

      There is Calcite rule JoinPushTransitivePredicatesRule but it does not work in Drill.
      Applying it in Drill will allow for equi-join queries to push filter condition from one table to another:

      select * 
from A, B 
where
A.id = B.id and 
B.id = 100

      In that case it is possible that Scan operator for A table will not scan all data.
      For table A it can lead for applying:
      1. Partition pruning for Hive tables and partiotion/directory pruning for file system tables
      2. Parquet filter pushdown

       

      Note: transitive closure doesn't work for some cases, these Calcite issues can resolve them:
      CALCITE-1048, CALCITE-2274, CALCITE-2275, CALCITE-2241. They are tracked by DRILL-6350

      Attachments

        Issue Links

          Dependency

          Improvement - An improvement or enhancement to an existing feature or task. DRILL-6350 Umbrella jira which tracks issues connected to Transitive Closure Inference

          • Major - Major loss of function.
          • Open
          depends upon

          Bug - A problem which impairs or prevents the functions of the product. CALCITE-1048 Make metadata more robust

          • Major - Major loss of function.
          • Open
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. CALCITE-2200 Infinite loop for JoinPushTransitivePredicatesRule

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. CALCITE-2205 JoinPushTransitivePredicatesRule should not create a Filter on top of an equivalent Filter

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. DRILL-6374 Transitive Closure leads to TPCH Queries regressions and OOM when run concurrency test

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. CALCITE-2275 Do not push down NOT condition in JOIN

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CALCITE-2274 Filter predicates aren't inferred while using dynamic star in subquery

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. CALCITE-1362 Filter aboved Join should consider the join key equivalence to push through

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CALCITE-2257 Combination of predicates can be proved to be always true

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. CALCITE-1299 Reference type of Pulled Up Predicates is not handled properly

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. CALCITE-2242 Using custom RelBuilder for FilterRemoveIsNotDistinctFromRule

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CALCITE-2241 Join equality condition should be considered while pushing down filter conditions with expressions disjunction for both tables

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #1216

          Activity

            People

              vitalii Vitalii Diravka
              vitalii Vitalii Diravka
              Aman Sinha Aman Sinha
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: