Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4879

Enhance Allocate Protocol to Identify Requests Explicitly

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      For legacy reasons, the current allocate protocol expects expanded requests which represent the cumulative request for any change in resource constraints. This is not only very difficult to comprehend but makes it impossible for the scheduler to associate container allocations to the original requests. This problem is amplified by the fact that the expansion is managed by the AMRMClient which makes it cumbersome for non-Java clients as they all have to replicate the non-trivial logic. In this JIRA, we are proposing enhancement to the Allocate Protocol to allow AMs to identify requests explicitly.

      Attachments

        1. SimpleAllocateProtocolProposal-v2.pdf
          189 kB
          Subramaniam Krishnan
        2. SimpleAllocateProtocolProposal-v1.pdf
          182 kB
          Subramaniam Krishnan

        Issue Links

          blocks

          Sub-task - The sub-task of the issue YARN-5415 Add support for NodeLocal and RackLocal OPPORTUNISTIC requests

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. YARN-5132 Exclude generated protobuf sources from YARN Javadoc build

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue YARN-314 Schedulers should allow resource requests of different sizes at the same priority and location

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-2915 Enable YARN RM scale out via federation using multiple RM's

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. REEF-1288 Add a request ID to resource requests / allocated Evaluators

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-4485 [Umbrella] Capture per-application and per-queue container allocation latency

          • Major - Major loss of function.
          • Open
          is related to

          Sub-task - The sub-task of the issue YARN-1042 add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue YARN-3870 Providing raw container request information for fine scheduling

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. YARN-110 AM releases too many containers due to the protocol

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-4902 [Umbrella] Generalized and unified scheduling-strategies in YARN

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-2877 Extend YARN to support distributed scheduling

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              subru Subramaniam Krishnan
              subru Subramaniam Krishnan
              Votes:
              2 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: