Description
cgroup v2 has some fundamental changes compared to v1. RHEL9, Ubuntu 22 already moved to cgroup v2 as a default, hence YARN should support it. This umbrella tracks the required work.
A way to test the newly added features:
- Turn on cgroup v1 based on the current documentation.
- System prerequisites:
- the file /etc/mtab should contain a mount path with the file system type cgroup2, by default this could be /sys/fs/cgroup on most OS's
- the cgroup.subtree_control file should contain the necessary controllers (update it with: echo "+cpu +io +memory" > cgroup.subtree_control)
- either create the YARN hierarchy and give recursive access to the user running the NM on the node. The hierarchy is hadoop-yarn by default (controller by yarn.nodemanager.linux-container-executor.cgroups.hierarchy), and recursive mode is required, because as soon as the directory is created it will be filled with the controller files which YARN will try to edit.
- Alternatively if the NM process user has access rights on the /sys/fs/cgroup directory it'll try to create the hierarchy and update the cgroup.subtree_control file.
- YARN configuration
- yarn.nodemanager.linux-container-executor.cgroups.mount-path should point to the directory where the cgroup2 structure is mounted and the hadoop-yarn hierarchy was created
- yarn.nodemanager.linux-container-executor.cgroups.v2.enabled should be set to true
- Enable a cgroup controller, like yarn. nodemanager. resource. cpu.enabled: true
- Launch the NM and monitor the cgroup files on container launches (i.e: /sys/fs/cgroup/hadoop-yarn/container_id/cpu.weight)
Attachments
1.
|
Update CGroupsBlkioResourceHandler implementation for cgroup v2 support | In Progress | Bence Kosztolnik | |
2.
|
Update OutboundBandwidthResourceHandler implementation for cgroup v2 support | Open | Unassigned | |
3.
|
Update CGroupElasticMemoryController for cgroup v2 support | Open | Unassigned | |
4.
|
Update GpuResourceHandler for cgroup v2 support | Open | Unassigned | |
5.
|
Update FpgaResourceHandler for cgroup v2 support | Open | Unassigned |