Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11285

LocalizedResources are leaked and its LocalPath are not cleared

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.2.1
    • None
    • nodemanager
    • None

    Description

      LocalizedResources are leaked and its LocalPath are not cleared from NM Local Directories.

      Each container has separate LocalizedResource object and separate local path like below.

         /mnt/yarn/usercache/hive/filecache/6/2552419:
         total 28456
         -r-x------ 1 yarn yarn 29135164 Aug  7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
      
         /mnt/yarn/usercache/hive/filecache/6/2552420:
         total 28456
         -r-x------ 1 yarn yarn 29135164 Aug  7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
      
         /mnt/yarn/usercache/hive/filecache/6/2552421:
         total 28456
         -r-x------ 1 yarn yarn 29135164 Aug  7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
      
         /mnt/yarn/usercache/hive/filecache/6/2552422:
         total 28456
      

      NM logs will be filled with below

      2022-08-07 09:00:00,275 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource (IPC Server handler 4 on 8040): Resource hdfs://hdfscluster/user/svc_di_data_eng/.hiveJars/hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar(->/mnt/yarn/usercache/data_eng_user/filecache/2498262/hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar) transitioned from LOCALIZED to null
      2022-08-07 09:00:00,340 INFO org.apache.hadoop.yarn.util.ProcfsBasedProcessTree (Container Monitor): SmapBasedCumulativeRssmem (bytes) : 0
      2022-08-07 09:00:00,386 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource (IPC Server handler 9 on 8040): Can't handle this event at current state
      org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: LOCALIZED at LOCALIZED
              at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
              at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
              at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource.handle(LocalizedResource.java:198)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:186)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:58)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.processHeartbeat(ResourceLocalizationService.java:1048)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:722)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:356)
              at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:48)
              at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:63)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
              at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
              at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
      

      Attachments

        Issue Links

          Activity

            People

              prabhujoseph Prabhu Joseph
              prabhujoseph Prabhu Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: