Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1492

truly shared cache for jars (jobjar/libjar)

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.4-alpha
    • 2.9.0, 3.0.0
    • None
    • None
    • Hide
      The YARN Shared Cache provides the facility to upload and manage shared application resources to HDFS in a safe and scalable manner. YARN applications can leverage resources uploaded by other applications or previous runs of the same application without having to re-­upload and localize identical files multiple times. This will save network resources and reduce YARN application startup time.
      Show
      The YARN Shared Cache provides the facility to upload and manage shared application resources to HDFS in a safe and scalable manner. YARN applications can leverage resources uploaded by other applications or previous runs of the same application without having to re-­upload and localize identical files multiple times. This will save network resources and reduce YARN application startup time.

    Description

      Currently there is the distributed cache that enables you to cache jars and files so that attempts from the same job can reuse them. However, sharing is limited with the distributed cache because it is normally on a per-job basis. On a large cluster, sometimes copying of jobjars and libjars becomes so prevalent that it consumes a large portion of the network bandwidth, not to speak of defeating the purpose of "bringing compute to where data is". This is wasteful because in most cases code doesn't change much across many jobs.

      I'd like to propose and discuss feasibility of introducing a truly shared cache so that multiple jobs from multiple users can share and cache jars. This JIRA is to open the discussion.

      Attachments

        1. shared_cache_design_v6.pdf
          251 kB
          Chris Trezzo
        2. YARN-1492-all-trunk-v5.patch
          456 kB
          Chris Trezzo
        3. YARN-1492-all-trunk-v4.patch
          455 kB
          Chris Trezzo
        4. YARN-1492-all-trunk-v3.patch
          455 kB
          Chris Trezzo
        5. YARN-1492-all-trunk-v2.patch
          455 kB
          Chris Trezzo
        6. YARN-1492-all-trunk-v1.patch
          435 kB
          Chris Trezzo
        7. shared_cache_design_v5.pdf
          253 kB
          Chris Trezzo
        8. shared_cache_design_v4.pdf
          25 kB
          Sangjin Lee
        9. shared_cache_design_v3.pdf
          26 kB
          Sangjin Lee
        10. shared_cache_design_v2.pdf
          26 kB
          Sangjin Lee
        11. shared_cache_design.pdf
          24 kB
          Sangjin Lee

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1901 Jobs should not submit the same jar files over and over again

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-5951 Add support for the YARN Shared Cache

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue YARN-1016 Define a HDFS based repository that allows YARN services to share resources

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-1020 Resource Localization using Groups as a new Localization Type

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-7282 Shared Cache Phase 2

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-1529 Add Localization overhead metrics to NM

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-1756 Capture one more timestamp for an application when ApplicationClientProtocol#getNewApplication is executed

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. PIG-2672 Optimize the use of DistributedCache

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. YARN-6117 SharedCacheManager does not start up

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. REEF-58 Add support for YARN SCM

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              ctrezzo Chris Trezzo
              sjlee0 Sangjin Lee
              Votes:
              3 Vote for this issue
              Watchers:
              81 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: