Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-684

Uber/Local modes for Tez

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • None
    • None

    Description

      Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.

      Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.

      Here is my design:

      Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.

      Attachments

        1. TezUberModeDesignDraft.png
          74 kB
          Chen He
        2. TEZ-684.patch
          151 kB
          Chen He
        3. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        4. TEZ-684.patch
          134 kB
          Jonathan Turner Eagles
        5. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        6. TEZ-684.patch
          121 kB
          Jonathan Turner Eagles
        7. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        8. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        9. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        10. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        11. TEZ-684.patch
          115 kB
          Chen He
        12. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        13. TEZ-684.patch
          142 kB
          Chen He
        14. TEZ-684-2014-7-21.patch
          118 kB
          Chen He

        Issue Links

          is blocked by

          Task - A task that needs to be done. TEZ-1343 Bypass the Fetcher and read directly from the local filesystem if source vertex ran on the same host

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. TEZ-235 Tez Local Mode

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-1844 Shouldn't invoke system.exit in local mode when AM is failed to start

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. PIG-4008 Pig code change to enable Tez Local mode

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. TEZ-1837 Restrict usage of Environment variables to main methods

          • Major - Major loss of function.
          • Closed
          requires

          Sub-task - The sub-task of the issue TEZ-1191 DAGAppMaster shuts down JVM resulting in shutdown propagation up the stack.

          • Major - Major loss of function.
          • Reopened
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. TEZ-72 Support running of single-task vertices inlined in the DAG AM

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              airbots Chen He
              airbots Chen He
              Votes:
              6 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: