Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3798

Race condition may cause scanners to spin with runtime filters on Avro or Sequence files

    XMLWordPrintableJSON

Details

    Description

      If a runtime filter arrives after per-file filtering is done, but before per-scan-range filtering, the filter may cause a header split from an Avro file to be filtered out. However, that means that BaseSequenceScanner::ProcessSplit() is not called, which would otherwise issue the remaining scan ranges in that file. This means that the exit condition for HdfsScanNode::ScannerThread() is not met, as num_unqueued_files_ and progress_ are never set to 0 or num_splits respectively. The scanner threads will therefore spin.

      Attachments

        Issue Links

          Activity

            People

              henryr Henry Robinson
              henryr Henry Robinson
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: