Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22008

writing metadata is not an atomic operation, we should add a commit logic

    XMLWordPrintableJSON

Details

    Description

      writing metadata is not an atomic operation, some logic can cause there is a metadata file in the checkpoint dir, but the data is corrupted if the jobmanager crash while writing the metadata. 

       

      So we should consider to add commit operation in the checkpoint storage location

      Attachments

        Issue Links

          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. FLINK-9043 Introduce a friendly way to resume the job from externalized checkpoints automatically

          • Not a Priority -
          • Reopened
          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-17808 Rename checkpoint meta file to "_metadata" until it has completed writing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-17808 Rename checkpoint meta file to "_metadata" until it has completed writing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Web Link GitHub Pull Request #15409

          Web Link GitHub Pull Request #15410

          Activity

            People

              Unassigned Unassigned
              zhoujira86 xiaogang zhou
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: