Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.2.0, 3.3.1
-
None
-
None
-
Spark 3.2.2
kubernetes v1.20.8
jdk1.8.0_162
Description
I have a test for spark on k8s that my job had completed, but some tmp dirs are still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is mount on local NVMe SSD.
The DiskBlockManager may be clear the dirs just one iteration , then the process be killed.
The shuffle data will be getting larger and fill the disk space.