Details
Major Description
When calling FileUtils.listFiles and a file is being removed from this or a subdirectory during the execution the following Error occurs on linux.
Stacktrace
Caused by: java.nio.file.NoSuchFileException: /** at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.base/java.nio.file.Files.readAttributes(Files.java:1764) at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373) at java.base/java.nio.file.Files.walkFileTree(Files.java:2761) at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076) at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)
Upgrade of apache commons-io
The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the library started using java.nio instead of java.io which might behave differently on unix, when a file is not existent during processing of directories.
Causing functionality in our implementation
In some part we have multithreaded access to the filesystem. A folder is accessed from multiple threads which access the folder with org.apache.commons.io.FileUtils#listFiles(java.io.File, org.apache.commons.io.filefilter.IOFileFilter, org.apache.commons.io.filefilter.IOFileFilter). These threads also move, delete and rename files from this directory. While this seems to be working on a windows machine, it is not the case on a Linux machine. In 2.8 apparently there was no issue with this.
In short: While a directory is being listed with FileUtils.listFiles(), a file is being deleted. When trying to read its file-attributes it stops because of a NoSuchFileException.
Other commons-io functionalities
We tried using PathUtils.walk(directory filters, 5, false).map(Path::toFile).collect(Collectors.toList()). This method was introduced with the same change.
This produced a similar stacktrace:
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /* at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87) at java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ... 47 more Caused by: java.nio.file.NoSuchFileException: /* at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.base/java.nio.file.Files.readAttributes(Files.java:1764) at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373) at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:83) ... 59 more
git commit
https://github.com/apache/commons-io/commit/4a514d3306b55b3667d1449ebd4cbe5f19dd7af0
Other Questions
- What was the reason to wrap all IOException into UncheckedIOException in those methods?