Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26949

Prevent "purge" to remove needed batch files in CompactibleFileStreamLog

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Structured Streaming
    • None

    Description

      I've seen couple of trials (in opened PRs, even I've also tried) which calls purge() in CompactibleFileStreamLog, but after looking at the codebase of CompactibleFileStreamLog, I've realized that purging latest compaction batch would break internal of CompactibleFileStreamLog and it throws IllegalStateException.

      Given that CompactibleFileStreamLog maintains the batches and purges according to its configuration, it would be safer to just rely on CompactibleFileStreamLog to purge and prevent calling `purge` outside of CompactibleFileStreamLog.

      Attachments

        Issue Links

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: