Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
None
-
None
-
None
-
None
Description
Diretory "/tmp/hadoop-${user}/dfs/tmp/tmp" is being filled with those kinfd of files: client-226966559287638337420857.tmp
I tried to look at the code and found:
DFSClient.java
src/java/org/apache/hadoop/dfs/DFSClient.java
private void closeBackupStream() throws IOException {...} /* Similar to closeBackupStream(). Theoritically deleting a file * twice could result in deleting a file that we should not. */ private void deleteBackupFile() {...} private File newBackupFile() throws IOException { String name = "tmp" + File.separator + "client-" + Math.abs(r.nextLong()); File result = dirAllocator.createTmpFileForWrite(name, 2 * blockSize, conf); return result; }
LocalDirAllocator
src/java/org/apache/hadoop/fs/LocalDirAllocator.java#AllocatorPerContext.java
/** Creates a file on the local FS. Pass size as -1 if not known apriori. We * round-robin over the set of disks (via the configured dirs) and return * a file on the first path which has enough space. The file is guaranteed * to go away when the JVM exits. */ public File createTmpFileForWrite(String pathStr, long size, Configuration conf) throws IOException { // find an appropriate directory Path path = getLocalPathForWrite(pathStr, size, conf); File dir = new File(path.getParent().toUri().getPath()); String prefix = path.getName(); // create a temp file on this directory File result = File.createTempFile(prefix, null, dir); result.deleteOnExit(); return result; }
First it seems to me it's a bit of a mess here I don't know if it's DFSClient.java#deleteBackupFile() or LocalDirAllocator#createTmpFileForWrite() {deleteOnExit(); ) who is call ... or both. Why not keep it dry and delete it only once.
But the most important is the "deleteOnExit();" since it mean if it is never restarted it will never delete files
Attachments
Attachments
Issue Links
- is part of
-
HADOOP-1707 Remove the DFS Client disk-based cache
- Closed