Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10475

Scale RM-NM heartbeat interval based on node utilization

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.10.1, 3.4.0
    • 3.2.2, 3.4.0, 3.3.1, 3.2.3
    • yarn
    • None

    Description

      Add the ability to scale the RM-NM heartbeat interval based on node cpu utilization compared to overall cluster cpu utilization.  If a node is over-utilized compared to the rest of the cluster, it's heartbeat interval slows down.  If it is under-utilized compared to the rest of the cluster, it's heartbeat interval speeds up.

      This is a feature we have been running with internally in production for several years.  It was developed by nroberts, based on the observation that larger faster nodes on our cluster were under-utilized compared to smaller slower nodes. 

      This feature is dependent on YARN-10450, which added cluster-wide utilization metrics.

      Attachments

        1. YARN-10475-branch-3.3.003.patch
          35 kB
          Jim Brennan
        2. YARN-10475-branch-3.2.003.patch
          35 kB
          Jim Brennan
        3. YARN-10475.003.patch
          35 kB
          Jim Brennan
        4. YARN-10475.002.patch
          32 kB
          Jim Brennan
        5. YARN-10475.001.patch
          31 kB
          Jim Brennan

        Activity

          People

            jbrennan Jim Brennan
            jbrennan Jim Brennan
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: