Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41946

DiskBlockManager can't clear all of the tmp dirs in k8s

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0, 3.3.1
    • None
    • Block Manager, Spark Core
    • None
    • Spark 3.2.2

      kubernetes v1.20.8 

      jdk1.8.0_162

    Description

      I have a test for spark on k8s  that my job had completed, but some tmp dirs are still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is mount on local NVMe SSD.

      The DiskBlockManager may be clear the dirs  just one iteration , then the process be killed.

      The shuffle data will be getting larger and   fill the disk space.

      Attachments

        Activity

          People

            Unassigned Unassigned
            lfspace Peter Tian
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: