Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-3918

Unified and pluggable URI fetching support.

    XMLWordPrintableJSON

Details

    • Epic
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • URI Fetcher

    Description

      Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor binary) for executors/tasks. A recent refactor (MESOS-336) added caching support to the fetcher. The recent work on filesystem isolation/unified containerizer (MESOS-2840) requires Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The natural question is: can we leverage the fetcher to fetch those filesystem images (and cache them accordingly)? Unfortunately, the existing fetcher interface is tightly coupled with CommandInfo::URIs for executors/tasks, making it very hard to be re used to fetch/cache filesystem images.

      Another motivation for the refactor is that we want to extend the fetcher to support more types of schemes. For instance, we want to support magnet URI to enable p2p fetching. This is in fact quite important for operating a large cluster (MESOS-3596). The goal here is to allow fetcher to be extended (e.g., using modules) so that operators can add custom fetching support.

      The main design goal here is to decouple artifacts fetching from artifacts cache management. We can make artifacts fetching extensible (e.g. to support p2p fetching), and solve the cache management part later.

      Attachments

        Issue Links

          Activity

            People

              jieyu Jie Yu
              jieyu Jie Yu
              Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: