Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25256

Savepoints do not work with ExternallyInducedSources

    XMLWordPrintableJSON

Details

    Description

      It is not possible to take a proper savepoint with ExternallyInducedSource or ExternallyInducedSourceReader (both legacy and FLIP-27 versions). The problem is that we're hardcoding CheckpointOptions in the triggerHook.

      The outcome of current state is that operators would try to take checkpoints in the checkpoint location whereas the CheckpointCoordinator will write metadata for those states in the savepoint location.

      Moreover the situation gets even weirder (I have not checked it entirely), if we have a mixture of ExternallyInducedSource(s) and regular sources. In such a case the location and format at which the state of a particular task is persisted depends on the order of barriers arrival. If a barrier from a regular source arrives last the task takes a savepoint, on the other hand if last barrier is from an externally induced source it will take a checkpoint.

      Attachments

        Issue Links

          Activity

            People

              arvid Arvid Heise
              dwysakowicz Dawid Wysakowicz
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: