Details
-
Improvement
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
1.8.0
Description
Description
The Jepsen YARN tests sporadically fail because TM containers are exceeding their virtual memory limits:
Closing TaskExecutor connection container_1541436244107_0001_01_000005 because: Container [pid=32403,containerID=container_1541436244107_0001_01_000005] is running beyond virtual memory limits. Current usage: 970.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing container.
By default YARN enforces a virtual memory limit of 2.1 times the requested physical memory. However, in my experiments, the virtual memory of a JVM process running the ClusterEntryPoint (without submitting job) is already in the region of 3.3 GB. Hence, the virtual memory enforcement should be disabled.
Acceptance Criteria
- yarn.nodemanager.vmem-check-enabled is false in yarn-site.xml
Attachments
Issue Links
- links to