Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-67

/tmp/hadoop-${user}/dfs/tmp/tmp/client-${long}.tmp is not cleanup correctly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • None
    • None
    • None
    • None

    Description

      Diretory "/tmp/hadoop-${user}/dfs/tmp/tmp" is being filled with those kinfd of files: client-226966559287638337420857.tmp

      I tried to look at the code and found:

      DFSClient.java

      src/java/org/apache/hadoop/dfs/DFSClient.java

      private void closeBackupStream() throws IOException {...}
      
      /* Similar to closeBackupStream(). Theoritically deleting a file
       * twice could result in deleting a file that we should not.
       */
      private void deleteBackupFile() {...}
      
      private File newBackupFile() throws IOException {
      String name = "tmp" + File.separator +
                           "client-" + Math.abs(r.nextLong());
      File result = dirAllocator.createTmpFileForWrite(name,
                                                             2 * blockSize,
                                                             conf);
      return result;
      }
      

      LocalDirAllocator

      src/java/org/apache/hadoop/fs/LocalDirAllocator.java#AllocatorPerContext.java

      /** Creates a file on the local FS. Pass size as -1 if not known apriori. We
       *  round-robin over the set of disks (via the configured dirs) and return
       *  a file on the first path which has enough space. The file is guaranteed
       *  to go away when the JVM exits.
       */
      public File createTmpFileForWrite(String pathStr, long size,
              Configuration conf) throws IOException {
      
      // find an appropriate directory
      Path path = getLocalPathForWrite(pathStr, size, conf);
      File dir = new File(path.getParent().toUri().getPath());
      String prefix = path.getName();
      
      // create a temp file on this directory
      File result = File.createTempFile(prefix, null, dir);
      result.deleteOnExit();
      return result;
      }
      

      First it seems to me it's a bit of a mess here I don't know if it's DFSClient.java#deleteBackupFile() or LocalDirAllocator#createTmpFileForWrite() {deleteOnExit(); ) who is call ... or both. Why not keep it dry and delete it only once.

      But the most important is the "deleteOnExit();" since it mean if it is never restarted it will never delete files

      Attachments

        1. patch-DFSClient-HADOOP-2561.diff
          1 kB
          Benjamin Francisoud

        Issue Links

          Activity

            People

              Unassigned Unassigned
              francisoud Benjamin Francisoud
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: