Description
Tez Fetcher currently leaks a codec pool (~32kb) for each partition fetched
This causes task failures due to the direct buffers allocated by the codec, which are not freed until a full GC (but whose allocation does not trigger GC).
It is possible to perform the entire shuffle operations without ever triggering a full GC, the following case
Container exited with a non-zero exit code 143
], AttemptID:attempt_1399351577718_2330_1_05_000020_3 Info:Container container_1399351577718_2330_01_000310 COMPLETED with diagnostics set to [Container [pid=1734,containerID=container_1399351577718_2330_01_000310] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 5.4 GB of 40 GB virtual memory used. Killing container.
container_1399351577718_2330_01_000365/ $ grep -ri "CodecPool.*brand-new" syslog* | wc -l
6988
That is approx ~436Mb leak on a JVM spun up with -Xmx3500m & 4096m container.
Attachments
Attachments
Issue Links
- is depended upon by
-
HIVE-7158 Use Tez auto-parallelism in Hive
-
- Closed
-