Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 3.1.0
-
None
-
ghx-label-1
Description
To limit the memory consumption of catalog, we should experiment with a mechanism automatically evicting recently unused tables from catalogd. Initial design:
- impalad to report periodically/asynchronously the set of catalog objects that were accessed
- catalogd to record some kind of last access time
- catalogd to have some facility to scan over all catalog objects, collect some number of not-recently-used ones (eg to reach a target amount of evicted memory), and issue invalidate commands to itself
- no need to have exact LRU behavior – to simplify, we probably shouldn't try to do a classical LRU linked list between all catalog objects.
- initial patch probably just triggered manually. Discussed either running this on a schedule or running this based on JMX GC notifications if we see that the catalogd finished an old-gen GC and the old gen is more than some target percentage full.
Attachments
Issue Links
- relates to
-
IMPALA-7614 Impala 3.1 Doc: Document the New Invalidate Options
- Closed