Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-31953 FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source
  3. FLINK-32019

EARLIEST offset strategy for partitions discoveried later based on FLIP-288

    XMLWordPrintableJSON

Details

    Description

      As described in FLIP-288, the strategy used for new partitions is the same as the initial offset strategy, which is not reasonable.

      According to the semantics, if the startup strategy is latest, the consumed data should include all data from the moment of startup, which also includes all messages from new created partitions. However, the latest strategy currently maybe used for new partitions, leading to the loss of some data (thinking a new partition is created and might be discovered by Kafka source several minutes later, and the message produced into the partition within the gap might be dropped if we use for example "latest" as the initial offset strategy).if the data from all new partitions is not read, it does not meet the user's expectations.

      Other ploblems see final Section of FLIP-288User specifies OffsetsInitializer for new partition .

      Therefore, it’s better to provide an EARLIEST strategy for later discovered partitions.

      Attachments

        Issue Links

          Activity

            People

              loserwang1024 Hongshun Wang
              loserwang1024 Hongshun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: