Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-684

Uber/Local modes for Tez

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • None
    • None

    Description

      Similarly to MapReduce Uber-mode in Yarn, we plan to create the Uber-mode for Tez. It runs all tasks in local in one process.

      Our target is to start DAGAppMaster in local JVM and let it run all tasks in one process.

      Here is my design:

      Once user submits a DAG, Tez starts a instance of DAGAppMaster. This DAGAppMaster will check TezConfiguration before instantiate ContainerLauncher. If "is_Uber" is true, DAGAppMaster creates a LocalContainerLauncher. LocalTaskScheduler and LocalTaskSchedulerEventHandler will call LocalContainerLauncher to run all tasks one by one in a single JVM. Communications between ResourceManager and local classes (DAGAppMaster, LocalContainerLauncher, LocalTaskScheduler, and LocalTaskSchedulerEventHandler) are muted.

      Attachments

        1. TEZ-684-2014-7-21.patch
          118 kB
          Chen He
        2. TEZ-684.patch
          142 kB
          Chen He
        3. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        4. TEZ-684.patch
          115 kB
          Chen He
        5. TEZ-684.patch
          119 kB
          Jonathan Turner Eagles
        6. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        7. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        8. TEZ-684.patch
          120 kB
          Jonathan Turner Eagles
        9. TEZ-684.patch
          121 kB
          Jonathan Turner Eagles
        10. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        11. TEZ-684.patch
          134 kB
          Jonathan Turner Eagles
        12. TEZ-684.patch
          125 kB
          Jonathan Turner Eagles
        13. TEZ-684.patch
          151 kB
          Chen He
        14. TezUberModeDesignDraft.png
          74 kB
          Chen He

        Issue Links

          is blocked by

          Task - A task that needs to be done. TEZ-1343 Bypass the Fetcher and read directly from the local filesystem if source vertex ran on the same host

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. TEZ-235 Tez Local Mode

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-1844 Shouldn't invoke system.exit in local mode when AM is failed to start

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. PIG-4008 Pig code change to enable Tez Local mode

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. TEZ-1837 Restrict usage of Environment variables to main methods

          • Major - Major loss of function.
          • Closed
          requires

          Sub-task - The sub-task of the issue TEZ-1191 DAGAppMaster shuts down JVM resulting in shutdown propagation up the stack.

          • Major - Major loss of function.
          • Reopened
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. TEZ-72 Support running of single-task vertices inlined in the DAG AM

          • Major - Major loss of function.
          • Resolved
          1.
          		 Create LocalContainerLauncher Sub-task Closed Chen He
          2.
          		 Avoid asking for new containers in uber-local mode Sub-task Closed Jonathan Turner Eagles
          3.
          		 LocalFetcher for the Uber-mode Sub-task Closed Chen He
          4.
          		 Get Tez Local-mode work Sub-task Resolved Chen He
          5.
          		 Client changes to allow local mode DAG submission Sub-task Closed Jonathan Turner Eagles
          6.
          		 LocalContainerLauncher improvement - error handling, multiple tasks, Tez events Sub-task Closed Siddharth Seth
          7.
          		 Tez Configuration changes for TezLocalRunner Sub-task Closed Chen He
          8.
          		 Add Utility for Modifying Environment Variables Sub-task Closed Jonathan Turner Eagles
          9.
          		 Add parameterized constructor capabilities in ReflectionUtils Sub-task Closed Jonathan Turner Eagles
          10.
          		 Create execution underlying framework abstraction layer Sub-task Closed Jonathan Turner Eagles
          11.
          		 Support submission of multiple applications with LocalRunner from within the same JVM Sub-task Closed Siddharth Seth
          12.
          		 Add documentation for LocalMode usage Sub-task Closed Siddharth Seth
          13.
          		 Session mode and ignore-lib-uris needs to be set for session mode Sub-task Closed Siddharth Seth
          14.
          		 Avoid sysexit in the DAGAM in case of local mode Sub-task Closed Siddharth Seth
          15.
          		 Additional supplement for TEZ local mode document Sub-task Closed Chen He
          16.
          		 Disable node blacklisting and ATS in AM for local mode Sub-task Closed Jonathan Turner Eagles
          17.
          		 Tez Local Mode fails if "fs.defaultFS" is not set to be "file:///' in core-site.xml Sub-task Resolved Chen He
          18.
          		 Local Mode crashes on concurrentModificationException Sub-task Closed Chen He
          19.
          		 Add multiDAG session test and move TestLocalMode to tez-tests Sub-task Closed Chen He
          20.
          		 LocalTaskSchedulerService.getTotalResources() and getAvailableResources() can get negative if JVM memory is larger than 2GB Sub-task Closed Chen He

          Activity

            People

              airbots Chen He
              airbots Chen He
              Votes:
              6 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: