Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18342

HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.0.1
    • 2.0.2, 2.1.0
    • Structured Streaming
    • None

    Description

      The HDFSBackedStateStore renames temporary files to delta files as it commits new versions. It however doesn't check whether the rename succeeded. If the rename fails, then recovery will not be possible. It should fail during the rename stage.

      Attachments

        Activity

          People

            brkyvz Burak Yavuz
            brkyvz Burak Yavuz
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: