[MAPREDUCE-4893] MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.0.3-alpha, 0.23.7
Component/s: applicationmaster
Labels:
None

Hadoop Flags:

Reviewed

Description

Say the MR AppMaster asks the RM for 3 containers on nodes n1, n2 and n3. There are 10 node n1-n10 in the same rack. The RM can give it allocated containers in the list order n5, n2, n1. The way AM map->container assignment happens, the AM will try to assign node local maps to n5, failing which it will assign rack local maps to n5. These rack local maps could be node local on n2 and n1 and would have been assigned to containers on n1 and n2 if the AM had not made an early rack local match for them on n5. This can lead to poor locality.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

MAPREDUCE-4893.1.patch
19/Dec/12 23:37
19 kB
Bikas Saha
MAPREDUCE-4893.2.patch
11/Jan/13 19:34
19 kB
Bikas Saha
MAPREDUCE-4893.3.patch
30/Jan/13 04:29
19 kB
Bikas Saha

Issue Links

is duplicated by

MAPREDUCE-4982 AM hung with one pending map task

Resolved

is related to

MAPREDUCE-5622 MRAppMaster doesn't assign all allocated NODE_LOCAL containers to node-local maps

Open

MAPREDUCE-5002 AM could potentially allocate a reduce container to a map attempt

Resolved

Activity

People

Assignee:: Bikas Saha

Reporter:: Bikas Saha

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 19/Dec/12 20:02

Updated:: 13/Nov/13 15:40

Resolved:: 31/Jan/13 00:30