[MAPREDUCE-463] The job setup and cleanup tasks should be optional - ASF JIRA

XML

Word

Printable

JSON

Hadoop Flags:

Reviewed
Release Note:

Hide
Added Configuration property "mapred.committer.job.setup.cleanup.needed" to specify whether job-setup and job-cleanup is needed for the job output committer. The default value is true.
Added Job.setJobSetupCleanupNeeded and JobContext.getJobSetupCleanupNeeded api for setting/getting the configuration.
If the configuration is set to false, no setup or cleanup will be done.

Show
Added Configuration property "mapred.committer.job.setup.cleanup.needed" to specify whether job-setup and job-cleanup is needed for the job output committer. The default value is true. Added Job.setJobSetupCleanupNeeded and JobContext.getJobSetupCleanupNeeded api for setting/getting the configuration. If the configuration is set to false, no setup or cleanup will be done.

For jobs that require low latency and do not require setup or cleanup tasks for the job, it should be possible to turn them off for that job.

relates to

MAPREDUCE-1099 Setup and cleanup tasks could affect job latency if they are caught running on bad nodes

MAPREDUCE-4488 Port MAPREDUCE-463 (The job setup and cleanup tasks should be optional) to branch-1

MAPREDUCE-308 A JobInProgressLIstener can change a job without informing other listeners