Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2727

ProbeFilters can be incorrectly disabled due to cardinality agnostic check

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • Impala 2.2
    • Impala 2.5.0
    • Backend

    Description

      The check to disable probe filter in

      HdfsParquetScanner::ScalarColumnReader::InitDataPage

      doesn't take into account the probe side cardinality.

          // Check if we should disable the bitmap filter. We'll do this if the filter
          // is not removing a lot of rows.
          // TODO: how to pick the selectivity?
          if (bitmap_filter_ != NULL && bitmap_filter_rows_processed_ > 10000 &&
              bitmap_filter_rows_rejected_ < bitmap_filter_rows_processed_ * .1) {
            bitmap_filter_ = NULL;
          }
      

      Also selectivity of 10% is rather low, we can bump this up to 20% after other BloomFilter code is in place.

      Attachments

        Issue Links

          Activity

            People

              jbapple Jim Apple
              mmokhtar Mostafa Mokhtar
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: