Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-945

Cleanup spillable map files eagerly as part of close

    XMLWordPrintableJSON

Details

    Description

      Currently, files used by external spillable map are deleted on exits. For spark-streaming/deltastreamer continuous-mode cases which runs several iterations, it is better to eagerly delete files on closing the handles using it. 

      We need to eagerly delete the files on following cases:

      1. HoodieMergeHandle
      2. HoodieMergedLogRecordScanner
      3. SpillableMapBasedFileSystemView

      Attachments

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. HUDI-1652 DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #2069

          Activity

            People

              shivnarayan sivabalan narayanan
              vbalaji Balaji Varadarajan
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: