Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.6.0, 0.5.3, 0.7.0
-
None
-
None
-
Reviewed
Description
The min-held containers constraint is respected during query idle times, but is not respected when a query is actually in motion.
The AM releases unused containers during dag execution without checking for min-held containers.
2015-03-20 15:41:53,475 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing container, containerId=container_1424502260528_1348_01_000013, containerExpiryTime=1426891313264, idleTimeoutMin=5000 2015-03-20 15:41:53,475 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: Releasing unused container: container_1424502260528_1348_01_000013
This is actually useful only after the AM has received a soft pre-emption message, doing it on an idle cluster slows down one of the most common query patterns in BI systems.
create temporary table smalltable as ...; select ... bigtable JOIN smalltable ON ...;
The smaller query in the beginning throws away the pre-warmed capacity.