Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3825

MR should not be getting duplicate tokens for a MR Job.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.23.1, 2.0.0-alpha
    • None
    • security
    • None

    Description

      This is the counterpart to HADOOP-7967.
      MR gets tokens for all input, output and the default filesystem when a MR job is submitted.

      The APIs in FileSystem make it challenging to avoid duplicate tokens when there are file systems that have embedded
      filesystems.

      Here is the original description that Daryn wrote:
      The token cache currently tries to assume a filesystem's token service key. The assumption generally worked while there was a one to one mapping of filesystem to token. With the advent of multi-token filesystems like viewfs, the token cache will try to use a service key (ie. for viewfs) that will never exist (because it really gets the mounted fs tokens).

      The descriop

      Attachments

        1. MAPREDUCE-3825.patch
          8 kB
          Daryn Sharp
        2. TokenCache.pdf
          44 kB
          Daryn Sharp
        3. solution4.patch
          13 kB
          Daryn Sharp

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-7967 Need generalized multi-token filesystem support

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-2854 SecurityUtil.buildTokenService returns java.net.UnknownHostException when using paths like viewfs://default/some/path

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-8048 Allow merging of Credentials

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-3849 Change TokenCache's reading of the binary token file

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-3850 Avoid redundant calls for tokens in TokenCache

          • Major - Major loss of function.
          • Closed

          Activity

            People

              daryn Daryn Sharp
              daryn Daryn Sharp
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: