Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4801

ShuffleHandler can generate large logs due to prematurely closed channels

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.23.3, 2.0.1-alpha
    • 2.0.3-alpha, 0.23.5
    • None
    • None

    Description

      We ran into an instance where many nodes on a cluster ran out of disk space because the nodemanager logs were huge. Examining the logs showed many, many shuffle errors due to either ClosedChannelException or IOException from "Connection reset by peer" or "Broken pipe".

      Attachments

        1. MAPREDUCE-4801.patch
          10 kB
          Jason Darrell Lowe

        Activity

          People

            jlowe Jason Darrell Lowe
            jlowe Jason Darrell Lowe
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: