Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3701

Race Condition between cleanup thread and download tasks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0, 2.1.0, 2.2.0
    • 2.3.0
    • storm-server
    • None

    Description

      We captured a issue on our supervisor:

      
      2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task Executor - 0 [INFO] completelyRemoveUnusedUser directu for directory /home/y/var/storm/supervisor/usercache/directu
       2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [WARN] Caught Exception While Downloading (rethrowing)...
       java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist
               at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) ~[storm-client-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) ~[storm-client-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) ~[storm-client-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698) [storm-server-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683) [storm-server-2.3.0.y.jar:2.3.0.y]
               at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) [?:1.8.0_242]
               at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
               at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242]
               at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242]
               at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_242]
               at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
               at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
       2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR] java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist
       2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782}
       2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base blobs have not been downloaded yet.
       java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist
               at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) ~[storm-client-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) ~[storm-client-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) ~[storm-client-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362) ~[storm-server-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472) [storm-server-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549) [storm-server-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298) [storm-server-2.3.0.y.jar:2.3.0.y]
               at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039) [storm-server-2.3.0.y.jar:2.3.0.y]
      
      

      The root issue is the delay at https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641 which will cause the safeTopologyIds information out-of-date.

       

       

       

      Attachments

        Activity

          People

            li530 Rui Li
            li530 Rui Li
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h 10m
                2h 10m