Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3060

Generic shuffle service

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.23.0
    • None
    • mrv2
    • mrv2, shuffle

    Description

      When I was talking to Owen about MAPREDUCE-2600, we came across (again, talked about it with Chris before) the shuffle dependency issue. NodeManager currently has an implicit (hidden by the service plugin mechanism) dependency of a specific version of mapreduce shuffle. While this works in many cases, as long as we don't change shuffle headers and the usage of mapred security tokens, it's a hack to make things work none the less. It's generally agreed upon that nodemanager should only load generic services that are mapreduce framework neutral.

      In this particular case, the right solution seems to be a generic shuffle handler that can serve data for a particular partition securely. The ShuffleHandler currently only depends on mapreduce for task tokens and shuffle header, which is only used for writing data, i.e., the shuffle handler has no semantic dependency on mapreduce.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              vicaya Luke Lu
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated: