Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
2.10.2, 3.3.4
-
None
-
None
Description
Currently org.apache.hadoop.mapreduce.Job exposes getTaskReports(TaskType) API to fetch task reports of either mapper or reducer. However for MR jobs with large number of tasks this causes OOM issues while fetching all task reports as seen with JHS (HistoryClientService.getTaskReports), HistoryClientService also exposes an API getTaskReport(TaskId) where a TaskId can be provided within the GetTaskReportRequest. org.apache.hadoop.mapreduce.Job can expose 2 API so that individual task report can be fetched after listing them from client side
- Job.getTasks(TaskType) -> List<TaskId> - This would return TaskId of all tasks with given Type to the client
- Job.getTaskReport(TaskId) -> TaskReport - This would return task report for single task to the client
For JHS since JobHistoryParser.parse already parses full history file by default and maintains the list of tasks within JobHistoryParser.JobInfo's tasksMap, this info should be easy to get
One additional thing that needs to be seen is if this can be supported for requests which are redirected to MRClientService (within MRAppMaster) for running jobs