Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3440

Shuffling to memory can get out-of-sync when fetching multiple compressed map outputs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.7.2, 0.9.0, 0.8.5
    • None
    • None

    Description

      Haven't verified yet but certainly looks like tez needs same fix as MAPREDUCE-5308 in IFile.

      Specifically saw this because downstream tasks were reporting enough fetch failures that long-running upstream tasks had to be re-run, which makes job run for much longer than it needs.

      Usually shows itself as an "Invalid map id" error on a multi-map fetch on part 2-n (i.e. never the first one).

      Attachments

        1. TEZ-3440.patch
          46 kB
          Nathan Roberts
        2. TEZ-3440-v1.patch
          46 kB
          Nathan Roberts

        Issue Links

          Activity

            People

              nroberts Nathan Roberts
              nroberts Nathan Roberts
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: