Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3611

Support Docker Containers In LinuxContainerExecutor

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • yarn

    Description

      Support Docker Containers In LinuxContainerExecutor

      LinuxContainerExecutor provides useful functionality today with respect to localization, cgroups based resource management and isolation for CPU, network, disk etc. as well as security with a well-defined mechanism to execute privileged operations using the container-executor utility. Bringing docker support to LinuxContainerExecutor lets us use all of this functionality when running docker containers under YARN, while not requiring users and admins to configure and use a different ContainerExecutor.

      There are several aspects here that need to be worked through :

      • Mechanism(s) to let clients request docker-specific functionality - we could initially implement this via environment variables without impacting the client API.
      • Security - both docker daemon as well as application
      • Docker image localization
      • Running a docker container via container-executor as a specified user
      • “Isolate” the docker container in terms of CPU/network/disk/etc
      • Communicating with and/or signaling the running container (ensure correct pid handling)
      • Figure out workarounds for certain performance-sensitive scenarios like HDFS short-circuit reads
      • All of these need to be achieved without changing the current behavior of LinuxContainerExecutor

      Attachments

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. SPARK-20277 Allow Spark on YARN to be launched with Docker

          • Major - Major loss of function.
          • Resolved
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. YARN-5209 Transmission ContainerExecutor Class Parameters By The Client

          • Major - Major loss of function.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue YARN-7221 Add security check for privileged docker container

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-7446 Docker container privileged mode and --user flag contradict each other

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-7516 Security check for trusted docker image

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Wish - General wishlist item. YARN-3201 add args for DistributedShell to specify a image for tasks that will run on docker

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue YARN-7430 Enable user re-mapping for Docker containers by default

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. AMBARI-17353 First class support for YARN hosted services

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. YARN-7677 Docker image cannot set HADOOP_CONF_DIR

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-8472 YARN Container Phase 2

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-2466 Umbrella issue for Yarn launched Docker Containers

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-3291 DockerContainerExecutor should run as a non-root user inside the container

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              sidharta-s Sidharta Seethana
              sidharta-s Sidharta Seethana
              Votes:
              5 Vote for this issue
              Watchers:
              89 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: