Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
3.2.1
-
None
-
None
Description
LocalizedResources are leaked and its LocalPath are not cleared from NM Local Directories.
Each container has separate LocalizedResource object and separate local path like below.
/mnt/yarn/usercache/hive/filecache/6/2552419: total 28456 -r-x------ 1 yarn yarn 29135164 Aug 7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar /mnt/yarn/usercache/hive/filecache/6/2552420: total 28456 -r-x------ 1 yarn yarn 29135164 Aug 7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar /mnt/yarn/usercache/hive/filecache/6/2552421: total 28456 -r-x------ 1 yarn yarn 29135164 Aug 7 10:24 hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar /mnt/yarn/usercache/hive/filecache/6/2552422: total 28456
NM logs will be filled with below
2022-08-07 09:00:00,275 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource (IPC Server handler 4 on 8040): Resource hdfs://hdfscluster/user/svc_di_data_eng/.hiveJars/hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar(->/mnt/yarn/usercache/data_eng_user/filecache/2498262/hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar) transitioned from LOCALIZED to null 2022-08-07 09:00:00,340 INFO org.apache.hadoop.yarn.util.ProcfsBasedProcessTree (Container Monitor): SmapBasedCumulativeRssmem (bytes) : 0 2022-08-07 09:00:00,386 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource (IPC Server handler 9 on 8040): Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: LOCALIZED at LOCALIZED at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource.handle(LocalizedResource.java:198) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:186) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.handle(LocalResourcesTrackerImpl.java:58) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.processHeartbeat(ResourceLocalizationService.java:1048) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.processHeartbeat(ResourceLocalizationService.java:722) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.heartbeat(ResourceLocalizationService.java:356) at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.service.LocalizationProtocolPBServiceImpl.heartbeat(LocalizationProtocolPBServiceImpl.java:48) at org.apache.hadoop.yarn.proto.LocalizationProtocol$LocalizationProtocolService$2.callBlockingMethod(LocalizationProtocol.java:63) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
Attachments
Issue Links
- duplicates
-
YARN-9527 Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file
- Resolved