Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14840

Tool to estimate resource requirements of an application pipeline based on prior executions

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0, 3.1.0
    • tools
    • None
    • Reviewed
    • The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.

    Description

      We have been working on providing SLAs for job execution on Hadoop. At high level this involves 2 parts: deriving the resource requirements of a job and guaranteeing the estimated resources at runtime. The YARN ReservationSystem (YARN-1051/YARN-2572/YARN-5326) enable the latter and in this JIRA, we propose to add a tool to Hadoop to predict the resource requirements of a job based on past executions of the job. The system (aka Morpheus) deep dive can be found in our OSDI'16 paper here.

      Attachments

        1. ResourceEstimator-design-v1.pdf
          349 kB
          Subramaniam Krishnan
        2. HADOOP-14840-v4.patch
          373 kB
          Rui Li
        3. HADOOP-14840-v3.patch
          367 kB
          Rui Li
        4. HADOOP-14840-v2.patch
          365 kB
          Rui Li
        5. HADOOP-14840-v1.patch
          344 kB
          Rui Li
        6. HADOOP-14840-branch-2-v4.patch
          373 kB
          Subramaniam Krishnan

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. YARN-5326 Support for recurring reservations in the YARN ReservationSystem

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-1051 YARN Admission Control/Planner: enhancing the resource allocation model with time.

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14991 Add missing figures to Resource Estimator tool

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Rui Li Rui Li
              subru Subramaniam Krishnan
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: