Uploaded image for project: 'Slider'
  1. Slider
  2. SLIDER-82

Support ANTI_AFFINITY_REQUIRED option

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Slider 0.91
    • appmaster
    • None
    • Slider September #1

    Description

      slider has an anti-affinity flag in roles (visible in resources.json?), which is ignored.

      YARN-1042 promises this for YARN, slider will need

      1. flag in resources.json
      2. use in container requests

      we may also want two policies: anti-affinity-desired, and -required. Then if required nodes get >1 container for the same component type on the same node, it'd have to request a new one and return the old one (Risk: getting the same one back).

      Attachments

        1. SLIDER-82.002.patch
          14 kB
          Rajesh Kartha

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue SLIDER-945 build up history of request/release/failures for each component and serve in the REST API (and web UI?)

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. YARN-1412 Allocating Containers on a particular Node in Yarn

          • Major - Major loss of function.
          • Open
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. SLIDER-823 Über-JIRA : placement phase 3

          • Major - Major loss of function.
          • Open
          is duplicated by

          New Feature - A new feature of the product, which has yet to be developed. SLIDER-726 Different instances of the same component should run on different nodes (anti-affinity)

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SLIDER-1034 TestAgentAAEcho failed with #of pending instances == 49

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue YARN-1042 add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. TWILL-87 Adding Container Placement control (Placement Policy API)

          • Major - Major loss of function.
          • Resolved
          1.
          		 build node map from yarn update reports; serve via REST/IPC Sub-task Resolved Steve Loughran

          0%
          Original Estimate - 2h
          Remaining Estimate - 2h
          2.
          		 Write mock/unit tests for AA placement Sub-task Resolved Steve Loughran  
          3.
          		 RoleHistory to (carefully) share RoleStatus instances with AppState Sub-task Resolved Steve Loughran  
          4.
          		 Implement sequential AA assignment without location constraints Sub-task Resolved Steve Loughran  
          5.
          		 Use nodemap to build up location restrictions on AA placement Sub-task Resolved Steve Loughran

          0%
          Original Estimate - 8h
          Remaining Estimate - 8h
          6.
          		 Add mock test to simulate AA failure and restart of AA request sequence Sub-task Resolved Steve Loughran

          100%
          Original Estimate - 2h
          Time Spent - 0.5h Time Not Required
          7.
          		 Add functional tests of AA placement Sub-task Resolved Steve Loughran  
          8.
          		 revisit why OutstandingRequest.buildContainerRequest() sets label==null on a placed request Sub-task Resolved Unassigned  
          9.
          		 AM web UI to show state of AA request Sub-task Resolved Steve Loughran  
          10.
          		 add minicluster test of AA placement Sub-task Resolved Steve Loughran

          0%
          Original Estimate - 2h
          Remaining Estimate - 2h
          11.
          		 add mock test of failure of AA container and re-request; fix any failures Sub-task Resolved Steve Loughran

          0%
          Original Estimate - 1h
          Remaining Estimate - 1h
          12.
          		 Mock AA test for nodemap not updated Sub-task Resolved Steve Loughran  
          13.
          		 add "nodemap" command to get the (JSON) nodemap of the YARN cluster Sub-task Resolved Steve Loughran  
          14.
          		 Update slider docs with coverage of AA placement Sub-task Resolved Steve Loughran

          0%
          Original Estimate - 3h
          Remaining Estimate - 3h
          15.
          		 AM can't make YarnClient calls on a secure cluster Sub-task Resolved Steve Loughran  
          16.
          		 review label logic in AA code Sub-task Resolved Steve Loughran  
          17.
          		 add regression test to verify latest role history format reload Sub-task Resolved Steve Loughran  
          18.
          		 rename NO_DATA_LOCALITY placement policy to ANYWHERE Sub-task Resolved Steve Loughran  

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              1 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 186h
                  186h
                  Remaining:
                  Remaining Estimate - 184h
                  184h
                  Logged:
                  Remaining Estimate - 184h Time Not Required
                  0.5h