Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23717

Leverage docker support in Hadoop 3

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Spark Core, YARN
    • None

    Description

      The introduction of docker support in Apache Hadoop 3 can be leveraged by Apache Spark for resolving multiple long standing shortcoming - particularly related to package isolation.
      It also allows for network isolation, where applicable, allowing for more sophisticated cluster configuration/customization.

      This jira will track the various tasks for enhancing spark to leverage container support.

      Attachments

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23718 Document using docker in host networking mode in hadoop 3

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23719 Use correct hostname in non-host networking mode in hadoop 3 docker support

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23720 Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23721 Enhance BlockManagerId to include container's underlying host machine hostname

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue YARN-7994 Add support for network-alias in docker run for user defined networks

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              mridulm80 Mridul Muralidharan
              Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated: