Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2877 Extend YARN to support distributed scheduling
  3. YARN-2884

Proxying all AM-RM communications

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      We introduce the notion of an RMProxy, running on each node (or once per rack). Upon start the AM is forced (via tokens and configuration) to direct all its requests to a new services running on the NM that provide a proxy to the central RM.

      This give us a place to:
      1) perform distributed scheduling decisions
      2) throttling mis-behaving AMs
      3) mask the access to a federation of RMs

      Attachments

        1. YARN-2884-V9.patch
          139 kB
          Kishore Chaliparambil
        2. YARN-2884-V8.patch
          136 kB
          Kishore Chaliparambil
        3. YARN-2884-V7.patch
          135 kB
          Kishore Chaliparambil
        4. YARN-2884-V6.patch
          134 kB
          Kishore Chaliparambil
        5. YARN-2884-V5.patch
          134 kB
          Kishore Chaliparambil
        6. YARN-2884-V4.patch
          134 kB
          Kishore Chaliparambil
        7. YARN-2884-V3.patch
          134 kB
          Kishore Chaliparambil
        8. YARN-2884-V2.patch
          134 kB
          Kishore Chaliparambil
        9. YARN-2884-V13.patch
          144 kB
          Kishore Chaliparambil
        10. YARN-2884-V12.patch
          144 kB
          Kishore Chaliparambil
        11. YARN-2884-V11.patch
          137 kB
          Kishore Chaliparambil
        12. YARN-2884-V10.patch
          138 kB
          Kishore Chaliparambil
        13. YARN-2884-V1.patch
          154 kB
          Kishore Chaliparambil

        Issue Links

          blocks

          Sub-task - The sub-task of the issue YARN-3666 Federation Intercepting and propagating AM- home RM communications

          • Major - Major loss of function.
          • Resolved
          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. YARN-4083 Add a discovery mechanism for the scheduler address

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-4117 End to end unit test with mini YARN cluster for AMRMProxy Service

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Sub-task - The sub-task of the issue YARN-3116 [Collector wireup] We need an assured way to determine if a container is an AM container on NM

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-3927 Make the NodeManager's ContainerManager pluggable

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue YARN-5411 Create a proxy chain for ApplicationClientProtocol in the Router

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-5833 Add validation to ensure default ports are unique in Configuration

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-1547 Prevent DoS of ApplicationMasterProtocol by putting in limits

          • Major - Major loss of function.
          • Patch Available

          New Feature - A new feature of the product, which has yet to be developed. YARN-2915 Enable YARN RM scale out via federation using multiple RM's

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-6355 [Umbrella] Preprocessor framework for AM and Client interactions with the RM

          • Major - Major loss of function.
          • Open

          Activity

            People

              kishorch Kishore Chaliparambil
              curino Carlo Curino
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: