[MAPREDUCE-4842] Shuffle race can hang reducer - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Blocker
Resolution: Fixed
Affects Version/s: 2.0.2-alpha, 0.23.5
Fix Version/s: 2.0.3-alpha, 0.23.6
Component/s: mrv2
Labels:
None

Target Version/s:

2.0.3-alpha, 0.23.6

Description

Saw an instance where the shuffle caused multiple reducers in a job to hang. It looked similar to the problem described in ~~MAPREDUCE-3721~~, where the fetchers were all being told to WAIT by the MergeManager but no merge was taking place.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

MAPREDUCE-4842-2.patch
20/Dec/12 17:27
16 kB
Jason Darrell Lowe
MAPREDUCE-4842.patch
04/Dec/12 13:49
6 kB
Arun Murthy
MAPREDUCE-4842.patch
05/Dec/12 02:09
14 kB
Jason Darrell Lowe
MAPREDUCE-4842.patch
05/Dec/12 15:51
15 kB
Arun Murthy
MAPREDUCE-4842.patch
05/Dec/12 21:44
15 kB
Jason Darrell Lowe
mapreduce-4842.patch
06/Dec/12 23:06
6 kB
Mariappan Asokan
mapreduce-4842.patch
18/Dec/12 20:18
6 kB
Mariappan Asokan
mapreduce-4842.patch
20/Dec/12 17:31
6 kB
Mariappan Asokan
mapreduce-4842.patch
20/Dec/12 18:43
15 kB
Mariappan Asokan
mapreduce-4842.patch
20/Dec/12 20:10
15 kB
Mariappan Asokan
mapreduce-4842.patch
21/Dec/12 00:50
15 kB
Mariappan Asokan

Issue Links

is duplicated by

MAPREDUCE-5423 Rare deadlock situation when reducers try to fetch map output

Resolved

is related to

TEZ-3293 Fetch failures can cause a shuffle hang waiting for memory merge that never starts

Closed

relates to

MAPREDUCE-3721 Race in shuffle can cause it to hang

Closed

Activity

People

Assignee:: Mariappan Asokan

Reporter:: Jason Darrell Lowe

Votes:: 0 Vote for this issue

Watchers:: 11 Start watching this issue

Dates

Created:: 03/Dec/12 21:09

Updated:: 08/Jun/16 20:21

Resolved:: 21/Dec/12 18:36