Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3350

Shuffle spills are not spilled to a container-specific directory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.7.1
    • None
    • None
    • None

    Description

      If a Tez task receives too much input data and needs to spill the inputs to disk it ends up using a path that is not container-specific. Therefore YARN will not automatically cleanup these files when the container exits as it should, and instead the files linger until the entire application completes.

      Attachments

        Issue Links

          Activity

            People

              abstractdog László Bodor
              jlowe Jason Darrell Lowe
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: