Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11669

[Umbrella] cgroup v2 support

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • 3.5.0
    • yarn
    • None
    • Reviewed

    Description

      cgroup v2 has some fundamental changes compared to v1. RHEL9, Ubuntu 22 already moved to cgroup v2 as a default, hence YARN should support it. This umbrella tracks the required work.

      Documentation

      A way to test the newly added features:

      1. Turn on cgroup v1 based on the current documentation.
      2. System prerequisites:
        1. the file /etc/mtab should contain a mount path with the file system type cgroup2, by default this could be /sys/fs/cgroup on most OS's
        2. the cgroup.subtree_control file should contain the necessary controllers (update it with: echo "+cpu +io +memory" > cgroup.subtree_control)
        3. either create the YARN hierarchy and give recursive access to the user running the NM on the node. The hierarchy is hadoop-yarn by default (controller by yarn.nodemanager.linux-container-executor.cgroups.hierarchy), and recursive mode is required, because as soon as the directory is created it will be filled with the controller files which YARN will try to edit.
          1. Alternatively if the NM process user has access rights on the /sys/fs/cgroup directory it'll try to create the hierarchy and update the cgroup.subtree_control file.
      3. YARN configuration
        1. yarn.nodemanager.linux-container-executor.cgroups.mount-path should point to the directory where the cgroup2 structure is mounted and the hadoop-yarn hierarchy was created
        2. yarn.nodemanager.linux-container-executor.cgroups.v2.enabled should be set to true
        3. Enable a cgroup controller, like yarn. nodemanager. resource. cpu.enabled: true
      4. Launch the NM and monitor the cgroup files on container launches (i.e: /sys/fs/cgroup/hadoop-yarn/container_id/cpu.weight)

      Attachments

        Activity

          People

            bteke Benjamin Teke
            bender Ferenc Erdelyi
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: