Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5951

Add support for the YARN Shared Cache

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0
    • None
    • Hide
      MapReduce support for the YARN shared cache allows MapReduce jobs to take advantage of additional resource caching. This saves network bandwidth between the job submission client as well as within the YARN cluster itself. This will reduce job submission time and overall job runtime.
      Show
      MapReduce support for the YARN shared cache allows MapReduce jobs to take advantage of additional resource caching. This saves network bandwidth between the job submission client as well as within the YARN cluster itself. This will reduce job submission time and overall job runtime.

    Description

      Implement the necessary changes so that the MapReduce application can leverage the new YARN shared cache (i.e. YARN-1492).

      Specifically, allow per-job configuration so that MapReduce jobs can specify which set of resources they would like to cache (i.e. jobjar, libjars, archives, files).

      Attachments

        1. MAPREDUCE-5951-trunk-021.patch
          109 kB
          Chris Trezzo
        2. MAPREDUCE-5951-trunk-020.patch
          90 kB
          Chris Trezzo
        3. MAPREDUCE-5951-trunk.019.patch
          78 kB
          Chris Trezzo
        4. MAPREDUCE-5951-Overview.001.pdf
          68 kB
          Chris Trezzo
        5. MAPREDUCE-5951-trunk.018.patch
          76 kB
          Chris Trezzo
        6. MAPREDUCE-5951-trunk.017.patch
          76 kB
          Chris Trezzo
        7. MAPREDUCE-5951-trunk.016.patch
          101 kB
          Chris Trezzo
        8. MAPREDUCE-5951-trunk-v15.patch
          75 kB
          Chris Trezzo
        9. MAPREDUCE-5951-trunk-v14.patch
          75 kB
          Chris Trezzo
        10. MAPREDUCE-5951-trunk-v13.patch
          72 kB
          Chris Trezzo
        11. MAPREDUCE-5951-trunk-v12.patch
          72 kB
          Chris Trezzo
        12. MAPREDUCE-5951-trunk-v11.patch
          74 kB
          Chris Trezzo
        13. MAPREDUCE-5951-trunk-v10.patch
          83 kB
          Chris Trezzo
        14. MAPREDUCE-5951-trunk-v9.patch
          86 kB
          Chris Trezzo
        15. MAPREDUCE-5951-trunk-v8.patch
          86 kB
          Chris Trezzo
        16. MAPREDUCE-5951-trunk-v7.patch
          83 kB
          Chris Trezzo
        17. MAPREDUCE-5951-trunk-v6.patch
          103 kB
          Chris Trezzo
        18. MAPREDUCE-5951-trunk-v5.patch
          90 kB
          Chris Trezzo
        19. MAPREDUCE-5951-trunk-v4.patch
          89 kB
          Chris Trezzo
        20. MAPREDUCE-5951-trunk-v3.patch
          89 kB
          Chris Trezzo
        21. MAPREDUCE-5951-trunk-v2.patch
          84 kB
          Chris Trezzo
        22. MAPREDUCE-5951-trunk-v1.patch
          81 kB
          Chris Trezzo

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue YARN-3637 Handle localization sym-linking correctly at the YARN level

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2217 Shared cache client side changes

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. YARN-1492 truly shared cache for jars (jobjar/libjar)

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6267 Refactor JobSubmitter#copyAndConfigureFiles into it's own class

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6113 An ability to override LocalResourceVisibility for public resources

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6824 TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6365 Refactor JobResourceUploader#uploadFilesInternal

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue YARN-5727 Improve YARN shared cache support for LinuxContainerExecutor

          • Major - Major loss of function.
          • Open

          Activity

            People

              ctrezzo Chris Trezzo
              ctrezzo Chris Trezzo
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: