Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
0.18.0
-
None
-
None
-
Incompatible change
-
Changed Map-Reduce framework to no longer create temporary task output directories for staging outputs if staging outputs isn't necessary. ${mapred.out.dir}/_temporary/_${taskid}
Description
The staging directory for task-outputs (i.e. ${mapred.out.dir}/temporary/${taskid}) should only be created when Maps produce output on HDFS, which usually isn't the case. This plays very badly with HDFS quotas and may lead to thousands of temp names in the FS namespace, there-by overhauling the quotas. IAC, it isn't good to needlessly create these directories.