Executors management seems to behave strangely.
After calling spark.stop()
See this:
19/03/21 09:51:39 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 332.
19/03/21 09:51:39 INFO DAGScheduler: Executor lost: 332 (epoch 446)
19/03/21 09:51:39 INFO BlockManagerMasterEndpoint: Trying to remove executor 332 from BlockManagerMaster.
19/03/21 09:51:39 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(332, data-10.bdp.gin.merck.com, 38713, None)
19/03/21 09:51:39 INFO BlockManagerMaster: Removed 332 successfully in removeExecutor
and few minutes later:
19/03/21 09:54:26 WARN HeartbeatReceiver: Removing executor 332 with no recent heartbeats: 173942 ms exceeds timeout 120000 ms
19/03/21 09:54:26 ERROR YarnClusterScheduler: Lost an executor 332 (already removed): Executor heartbeat timed out after 173942 ms
19/03/21 09:54:26 INFO YarnClusterSchedulerBackend: Requesting to kill executor(s) 332
19/03/21 09:54:26 WARN YarnClusterSchedulerBackend: Executor to kill 332 does not exist!
19/03/21 09:54:26 INFO YarnClusterSchedulerBackend: Actual list of executor(s) to be killed is
can you upload the log file?