Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4834

All operators should operate within a memory constraint

    XMLWordPrintableJSON

Details

    • Epic
    • Status: Open
    • Major
    • Resolution: Unresolved
    • Impala 2.9.0
    • None
    • Backend
    • Operator memory constraints
    • ghx-label-9

    Description

      Once the reservation management of IMPALA-3200 is enabled, we should start modifying more operators to operate within a memory constraint, so that once the query starts the operators can run to completion.

      This means that the bulk of the operator memory, particularly the memory that depends on the input size, should be allocated from the buffer pool. The operator should then be able to execute so long as it gets it's minimum reservation.

      It is ok for now if expr memory and smaller amounts of memory like rowbatches and small decompression buffers that are streamed through the plan isn't included in this.

      We also don't need to handle arbitrarily sized rows, only up to a configurable threshold (e.g. a query option)

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue IMPALA-5485 Exchg / data-stream recv-side buffers should operate under a memory reservation

          • Major - Major loss of function.
          • Open
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-4679 Impala to set query mem_limit automatically for compute stats

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5701 Additional mem_limit settings for resource management (min mem, max before spill, max spill)

          • Major - Major loss of function.
          • Closed
          is related to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. IMPALA-6460 More flexible memory-based admission control policies

          • Major - Major loss of function.
          • Resolved

          Test - A new unit, integration or system test. IMPALA-7648 Add tests for all cases where OOM is expected

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5165 Allocate memory for all data from BufferPool

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              tarmstrong Tim Armstrong
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: