Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-17674

New consumer reset positions for newly added partitions before retrieving committed offsets

    XMLWordPrintableJSON

Details

    Description

      The update fetch positions flow in new consumer attempts to :

      1. retrieve committed offsets
      2. reset positions for partitions that may still require one.

      In the case that a partition with committed offsets is assigned in-between 1 and 2, the new consumer mistakenly included the newly added partition in the LIST_OFFSETS request issued as part of 2 (so it updates the new partition position using the partition offset and not the committed offsets)

      We should ensure that we preserve the initial set of partitions we're initializing, so that the reset excludes partitions that may require a positions but that are not in the initial set. Those partitions should get a position on the next poll iteration (going through steps 1 and 2)

      Attachments

        Activity

          People

            lianetm Lianet Magrans
            lianetm Lianet Magrans
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: