Details
-
Technical Debt
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.16.0
-
None
-
None
Description
We introduce new interfaces for local cleanup and global cleanup in FLINK-25432. The initial plan was to call them independently of each other. The motivation behind it was to allow independent cleanup logic (e.g. in the DefaultJobGraphStore for releasing and deleting the data from the StateHandleStore).
This can be improved by adding an order to cleanup and calling the local cleanup on each resource before calling the corresponding global cleanup in the Dispatcher.
This works under the assumption that it's ok to keep the resource in an inconsistent state during cleanup (i.e. it's ok for the resource to be in a state were the local artifacts are already cleaned but the global artifacts are not). That holds true because of how the Dispatcher interacts with the JobResultStore. It functions as a transaction manager for the cleanup here.
Attachments
Issue Links
- is caused by
-
FLINK-25432 Introduce common interfaces for cleaning up local and global job data
- Resolved
- is related to
-
FLINK-26117 JobManagerMetricGroup needs to implement GloballyCleanableResource as well
- Resolved