Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1020

FileBasedSink fails to remove temporary directory on Windows operating system

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Duplicate
    • Not applicable
    • 0.4.0
    • sdk-java-core

    Description

      Beam fails to remove temporary directory after completing pipelines and sinking output results on Windows operating system and throws the following exception

       
      Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
      	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
      	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
      	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
      	at org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
      Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
      	at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
      	at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
      	at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
      	at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
      	at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
      	at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
      	at org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
      	at org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
      	at org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
      	at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
      

      This is due to the use of wildcard * when matching all files in the temporary directory, while Windows does not support wildcard * in the file path.

      Attachments

        Issue Links

          Activity

            People

              tenghuanhe Tenghuan He
              tenghuanhe Tenghuan He
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 12h
                  12h
                  Remaining:
                  Remaining Estimate - 12h
                  12h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified