Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-17808

Rename checkpoint meta file to "_metadata" until it has completed writing

    XMLWordPrintableJSON

Details

    Description

      In practice, some developers or customers would use some strategy to find the recent _metadata as the checkpoint to recover (e.g as many proposals in FLINK-9043 suggest). However, there existed a "_meatadata" file does not mean the checkpoint have been completed as the writing to create the "_meatadata" file could break as some force quit (e.g. yarn application -kill).

      We could create the checkpoint meta stream to write data to file named as "_metadata.inprogress" and renamed it to "_metadata" once completed writing. By doing so, we could ensure the "_metadata" is not broken.

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. FLINK-9043 Introduce a friendly way to resume the job from externalized checkpoints automatically

          • Not a Priority -
          • Reopened

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-9325 generate the _meta file for checkpoint only when the writing is truly successful

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-22008 writing metadata is not an atomic operation, we should add a commit logic

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-22008 writing metadata is not an atomic operation, we should add a commit logic

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Web Link GitHub Pull Request #18157

          Activity

            People

              zuston Junfan Zhang
              yunta Yun Tang
              Votes:
              2 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: