Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.0.0, 2.1.0, 2.2.0
-
None
Description
We captured a issue on our supervisor:
2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task Executor - 0 [INFO] completelyRemoveUnusedUser directu for directory /home/y/var/storm/supervisor/usercache/directu 2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [WARN] Caught Exception While Downloading (rethrowing)... java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) ~[storm-client-2.3.0.y.jar:2.3.0.y] at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) ~[storm-client-2.3.0.y.jar:2.3.0.y] at org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) ~[storm-client-2.3.0.y.jar:2.3.0.y] at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698) [storm-server-2.3.0.y.jar:2.3.0.y] at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683) [storm-server-2.3.0.y.jar:2.3.0.y] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) [?:1.8.0_242] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] 2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR] java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist 2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782} 2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base blobs have not been downloaded yet. java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y] at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) ~[storm-client-2.3.0.y.jar:2.3.0.y] at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) ~[storm-client-2.3.0.y.jar:2.3.0.y] at org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) ~[storm-client-2.3.0.y.jar:2.3.0.y] at org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362) ~[storm-server-2.3.0.y.jar:2.3.0.y] at org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472) [storm-server-2.3.0.y.jar:2.3.0.y] at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549) [storm-server-2.3.0.y.jar:2.3.0.y] at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298) [storm-server-2.3.0.y.jar:2.3.0.y] at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039) [storm-server-2.3.0.y.jar:2.3.0.y]
The root issue is the delay at https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641 which will cause the safeTopologyIds information out-of-date.