[TEZ-1942] Number of tasks show in Tez UI with auto-reduce parallelism is misleading - ASF JIRA

XML

Word

Printable

JSON

Ran a simple hive query (with tez) and "--hiveconf hive.tez.auto.reducer.parallelism=true" . This internally turns on tez's auto reduce parallelism.

Job started off with 1009 reduce tasks
Tez reduces the number of reducers to 253
Job completes successfully, but TEZ UI shows 1009 as the number of reducers (and 253 tasks as successful tasks). This can be a little misleading.

I will attach the screenshots soon.

TEZ-1942.3.patch
16/Jan/15 13:44
20 kB
Prakash Ramachandran
TEZ-1942.2.patch
15/Jan/15 06:55
7 kB
Prakash Ramachandran
TEZ-1942.1.patch
15/Jan/15 03:29
5 kB
Prakash Ramachandran
TEZ-1942.1.branch-0.5.patch
19/Jan/15 13:24
19 kB
Prakash Ramachandran
Screen Shot 2015-01-14 at 9.18.54 AM.png
14/Jan/15 03:53
87 kB
Rajesh Balamohan
Screen Shot 2015-01-14 at 9.18.21 AM.png
14/Jan/15 03:53
101 kB
Rajesh Balamohan
result_with_primary_filter.png
14/Jan/15 14:16
137 kB
Prakash Ramachandran
result_with_direct_vertex.png
14/Jan/15 14:16
97 kB
Prakash Ramachandran
output.json
14/Jan/15 06:56
22 kB
Rajesh Balamohan