Description
cgroup v2 has some fundamental changes compared to v1. RHEL9, Ubuntu 22 already moved to cgroup v2 as a default, hence YARN should support it. This umbrella tracks the required work.
A way to test the newly added features:
- Turn on cgroup v1 based on the current documentation.
- System prerequisites:
- the file /etc/mtab should contain a mount path with the file system type cgroup2, by default this could be /sys/fs/cgroup on most OS's
- the cgroup.subtree_control file should contain the necessary controllers (update it with: echo "+cpu +io +memory" > cgroup.subtree_control)
- either create the YARN hierarchy and give recursive access to the user running the NM on the node. The hierarchy is hadoop-yarn by default (controller by yarn.nodemanager.linux-container-executor.cgroups.hierarchy), and recursive mode is required, because as soon as the directory is created it will be filled with the controller files which YARN will try to edit.
- Alternatively if the NM process user has access rights on the /sys/fs/cgroup directory it'll try to create the hierarchy and update the cgroup.subtree_control file.
- YARN configuration
- yarn.nodemanager.linux-container-executor.cgroups.mount-path should point to the directory where the cgroup2 structure is mounted and the hadoop-yarn hierarchy was created
- yarn.nodemanager.linux-container-executor.cgroups.v2.enabled should be set to true
- Enable a cgroup controller, like yarn. nodemanager. resource. cpu.enabled: true
- Launch the NM and monitor the cgroup files on container launches (i.e: /sys/fs/cgroup/hadoop-yarn/container_id/cpu.weight)