Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-74

Use anti-affinity in YARN container requests

    XMLWordPrintableJSON

Details

    • Wish
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.6.0
    • None
    • yarn
    • None

    Description

      Samza's YARN containers are currently allocated according to whatever default strategy the YARN RM uses. Sometimes, we get all of our containers allocated to a single NM, even when many NMs have available resources.

      YARN is in the process of getting an anti-affinity feature (YARN-1042). When this is implemented, we should consider using it to prevent all the containers from ending up on a single host.

      NOTE: This anti-affinity feature might conflict with a colocate-processor-with-data feature, if we were trying to colocate stream processors near their data.

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue YARN-1042 add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SAMZA-124 Provide AM based antiaffinity to avoid clumping of containers

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              criccomini Chris Riccomini
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: