Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-26175

Make local cleanup being called in global cleanup situations

    XMLWordPrintableJSON

Details

    • Technical Debt
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.16.0
    • None
    • Runtime / Coordination
    • None

    Description

      We introduce new interfaces for local cleanup and global cleanup in FLINK-25432. The initial plan was to call them independently of each other. The motivation behind it was to allow independent cleanup logic (e.g. in the DefaultJobGraphStore for releasing and deleting the data from the StateHandleStore).

      This can be improved by adding an order to cleanup and calling the local cleanup on each resource before calling the corresponding global cleanup in the Dispatcher.

      This works under the assumption that it's ok to keep the resource in an inconsistent state during cleanup (i.e. it's ok for the resource to be in a state were the local artifacts are already cleaned but the global artifacts are not). That holds true because of how the Dispatcher interacts with the JobResultStore. It functions as a transaction manager for the cleanup here.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mapohl Matthias Pohl
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: