Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-1602

Blobstore UTs are failing on Windows

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.0.0
    • 1.0.0
    • storm-core
    • None
    • Windows

    Description

      Blobstore related UTs are failed on Windows.

      -------------------------------------------------------------------------------
      Test set: org.apache.storm.blobstore.BlobStoreTest
      -------------------------------------------------------------------------------
      Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 2.306 sec <<< FAILURE! - in org.apache.storm.blobstore.BlobStoreTest
      testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest)  Time elapsed: 1.798 sec  <<< ERROR!
      java.nio.file.AccessDeniedException: D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\1457050287771.tmp -> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
      	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
      	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
      	at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
      	at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
      	at java.nio.file.Files.move(Files.java:1347)
      	at org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
      	at org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
      	at org.apache.storm.blobstore.BlobStoreTest.testMultiple(BlobStoreTest.java:397)
      	at org.apache.storm.blobstore.BlobStoreTest.testMultipleLocalFs(BlobStoreTest.java:168)
      
      testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest)  Time elapsed: 1.8 sec  <<< ERROR!
      java.io.IOException: Unable to delete file: target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
      
      testGetFileLength(org.apache.storm.blobstore.BlobStoreTest)  Time elapsed: 0.067 sec  <<< ERROR!
      java.io.IOException: Unable to delete file: target\blob-store-test-b6d39145-11ea-4aa7-ae30-28bda603fb3a\blobs\1017\data_test\data
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
      
      testBasicLocalFs(org.apache.storm.blobstore.BlobStoreTest)  Time elapsed: 0.124 sec  <<< ERROR!
      java.lang.RuntimeException: java.nio.file.DirectoryNotEmptyException: D:\storm\storm-core\target\blob-store-test-b74e39dc-8ce7-4c39-ab7a-e04a8519eb61\blobs\1017\data_test
      	at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:265)
      	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
      	at java.nio.file.Files.deleteIfExists(Files.java:1118)
      	at org.apache.storm.blobstore.FileBlobStoreImpl.delete(FileBlobStoreImpl.java:239)
      	at org.apache.storm.blobstore.FileBlobStoreImpl.deleteKey(FileBlobStoreImpl.java:178)
      	at org.apache.storm.blobstore.LocalFsBlobStore.deleteBlob(LocalFsBlobStore.java:226)
      	at org.apache.storm.blobstore.LocalFsBlobStore$$EnhancerByMockitoWithCGLIB$$8252dffa.CGLIB$deleteBlob$5(<generated>)
      	at org.apache.storm.blobstore.LocalFsBlobStore$$EnhancerByMockitoWithCGLIB$$8252dffa$$FastClassByMockitoWithCGLIB$$bca8d941.invoke(<generated>)
      	at org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
      	at org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
      	at org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
      	at org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27)
      	at org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:108)
      	at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36)
      	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:93)
      	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
      	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
      	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
      	at org.apache.storm.blobstore.LocalFsBlobStore$$EnhancerByMockitoWithCGLIB$$8252dffa.deleteBlob(<generated>)
      	at org.apache.storm.blobstore.BlobStoreTest.testBasic(BlobStoreTest.java:312)
      	at org.apache.storm.blobstore.BlobStoreTest.testBasicLocalFs(BlobStoreTest.java:163)
      
      testBasicLocalFs(org.apache.storm.blobstore.BlobStoreTest)  Time elapsed: 0.124 sec  <<< ERROR!
      java.io.IOException: Unable to delete file: target\blob-store-test-b74e39dc-8ce7-4c39-ab7a-e04a8519eb61\blobs\1017\data_test\data
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
      
      testLocalFsWithAuth(org.apache.storm.blobstore.BlobStoreTest)  Time elapsed: 0.314 sec  <<< ERROR!
      java.nio.file.AccessDeniedException: D:\storm\storm-core\target\blob-store-test-77e698b7-a4bc-4719-b33e-9a8b588fc2e3\blobs\1017\data_test\1457050288321.tmp -> D:\storm\storm-core\target\blob-store-test-77e698b7-a4bc-4719-b33e-9a8b588fc2e3\blobs\1017\data_test\data
      	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
      	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
      	at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
      	at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
      	at java.nio.file.Files.move(Files.java:1347)
      	at org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
      	at org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
      	at org.apache.storm.blobstore.BlobStoreTest.testWithAuthentication(BlobStoreTest.java:246)
      	at org.apache.storm.blobstore.BlobStoreTest.testLocalFsWithAuth(BlobStoreTest.java:158)
      
      testLocalFsWithAuth(org.apache.storm.blobstore.BlobStoreTest)  Time elapsed: 0.314 sec  <<< ERROR!
      java.io.IOException: Unable to delete file: target\blob-store-test-77e698b7-a4bc-4719-b33e-9a8b588fc2e3\blobs\1017\data_test\data
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
      	at org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
      
      -------------------------------------------------------------------------------
      Test set: org.apache.storm.localizer.LocalizerTest
      -------------------------------------------------------------------------------
      Tests run: 13, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.23 sec <<< FAILURE! - in org.apache.storm.localizer.LocalizerTest
      testArchivesTar(org.apache.storm.localizer.LocalizerTest)  Time elapsed: 0.037 sec  <<< FAILURE!
      java.lang.AssertionError: blob uncompressed doesn't contain symlink
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.assertTrue(Assert.java:41)
      	at org.apache.storm.localizer.LocalizerTest.testArchives(LocalizerTest.java:295)
      	at org.apache.storm.localizer.LocalizerTest.testArchivesTar(LocalizerTest.java:252)
      
      testArchivesTgz(org.apache.storm.localizer.LocalizerTest)  Time elapsed: 0.012 sec  <<< FAILURE!
      java.lang.AssertionError: blob uncompressed doesn't contain symlink
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.assertTrue(Assert.java:41)
      	at org.apache.storm.localizer.LocalizerTest.testArchives(LocalizerTest.java:295)
      	at org.apache.storm.localizer.LocalizerTest.testArchivesTgz(LocalizerTest.java:237)
      
      testArchivesTarGz(org.apache.storm.localizer.LocalizerTest)  Time elapsed: 0.013 sec  <<< FAILURE!
      java.lang.AssertionError: blob uncompressed doesn't contain symlink
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.assertTrue(Assert.java:41)
      	at org.apache.storm.localizer.LocalizerTest.testArchives(LocalizerTest.java:295)
      	at org.apache.storm.localizer.LocalizerTest.testArchivesTarGz(LocalizerTest.java:247)
      

      And blobstore also makes supervisor crashed on grouping-test.

      20839 [Thread-13] INFO  o.a.s.d.supervisor - Downloading code for storm id topologytest-ecc87348-ca6d-4b0a-a836-4ad3b60f7b64-1-0
      20843 [Thread-13] INFO  o.a.c.f.i.CuratorFrameworkImpl - Starting
      20848 [Thread-13] INFO  o.a.s.b.FileBlobStoreImpl - Creating new blob store based in D:\tmp\5e742303-a2a4-4880-a5f9-cdaf9551ad11\blobs
      20933 [Thread-13] ERROR o.a.s.event - Error when processing event
      java.io.IOException: Unable to delete file: D:\tmp\5d3907f5-513a-4b86-8f21-877ab3425ba1\supervisor\tmp\64ac7267-6837-4a59-95cf-afb74403d334\stormconf.ser
      	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279) ~[commons-io-2.4.jar:2.4]
      	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653) ~[commons-io-2.4.jar:2.4]
      	at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535) ~[commons-io-2.4.jar:2.4]
      	at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:2756) ~[commons-io-2.4.jar:2.4]
      	at org.apache.storm.daemon.supervisor$fn__9646.invoke(supervisor.clj:1160) ~[classes/:?]
      	at clojure.lang.MultiFn.invoke(MultiFn.java:243) ~[clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9365$fn__9383.invoke(supervisor.clj:571) ~[classes/:?]
      	at org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9365.invoke(supervisor.clj:570) ~[classes/:?]
      	at org.apache.storm.event$event_manager$fn__8912.invoke(event.clj:40) [classes/:?]
      	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
      	at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80-ea]
      20957 [Thread-13] ERROR o.a.s.util - Halting process: ("Error when processing an event")
      java.lang.RuntimeException: ("Error when processing an event")
      	at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [classes/:?]
      	at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
      	at org.apache.storm.event$event_manager$fn__8912.invoke(event.clj:48) [classes/:?]
      	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
      	at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80-ea]
      

      After failed UT, I can see stormconf.ser at that location, and also able to delete file via 'del' from command line.

      Attachments

        Issue Links

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: