[FLINK-34255] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: In Progress
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: 1.20.0, 2.0-preview
Component/s: Runtime / Checkpointing, Runtime / State Backends
Labels:
None

Release Note:

Hide
Currently all the options about state and checkpointing are reorganized and categorized by prefixes as listed below:

1. execution.checkpointing: all configurations associated with checkpointing and savepoint.
2. execution.state-recovery: all configurations pertinent to state recovery.
3. state.*: all configurations related to the state accessing.
    1. state.backend.*: specific options for individual state backends, such as RocksDB.
    2. state.changelog: configurations for the changelog, as outlined in FLIP-158, including the options for the "Durable Short-term Log" (DSTL).
    3. state.latency-track: configurations related to the latency tracking of state access.

At the meantime, all the original options scattered everywhere are annotated as @Deprecated

Show
Currently all the options about state and checkpointing are reorganized and categorized by prefixes as listed below: 1. execution.checkpointing: all configurations associated with checkpointing and savepoint. 2. execution.state-recovery: all configurations pertinent to state recovery. 3. state.*: all configurations related to the state accessing.     1. state.backend.*: specific options for individual state backends, such as RocksDB.     2. state.changelog: configurations for the changelog, as outlined in FLIP-158, including the options for the "Durable Short-term Log" (DSTL).     3. state.latency-track: configurations related to the latency tracking of state access. At the meantime, all the original options scattered everywhere are annotated as @Deprecated

Description

The FLIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560

Currently, the configuration options pertaining to checkpointing, recovery, and state management are primarily grouped under the following prefixes:

state.backend.* : configurations related to state accessing and checkpointing, as well as specific options for individual state backends
execution.checkpointing.* : configurations associated with checkpoint execution and recovery
execution.savepoint.{*}: configurations for recovery from savepoint

In addition, there are several individual options such as state.checkpoint-storage and state.checkpoints.dir that fall outside of these prefixes. The current arrangement of these options, which span multiple modules, is somewhat haphazard and lacks a systematic structure. For example, the options under the {{CheckpointingOptions }}and {{ExecutionCheckpointingOptions }}are related and have no clear boundaries from the user's perspective, but there is no unified prefix for them. With the upcoming release of Flink 2.0, we have an excellent opportunity to overhaul and restructure the configurations related to checkpointing, recovery, and state management. This FLIP proposes to reorganize these settings, making it more coherent by module, which would significantly lower the barriers for understanding and reduce the development costs moving forward.

Attachments

Issue Links

mentioned in: Page Loading...; Page Loading...

Sub-Tasks

There are no Sub-Tasks for this issue.

Activity

People

Assignee:: Zakelly Lan

Reporter:: Zakelly Lan

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 29/Jan/24 05:47

Updated:: 12/Sep/24 03:07