Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.4.1
-
None
-
Reviewed
Description
I wrote a standalone benchmark of the MapOutputBuffer and found that it did a lot of allocations during the merge phase. After looking at an allocation profile, I found that IFile.Reader.nextRawValue() would always allocate a new byte array for every value, so the allocation rate goes way up during the merge phase of the mapper. I imagine this also affects the reducer input, though I didn't profile that.
Attachments
Attachments
Issue Links
- relates to
-
HADOOP-5494 IFile.Reader should have a nextRawKey/nextRawValue
- Closed