Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-17505

New consumer seekToBeginning/End should run in background thread

    XMLWordPrintableJSON

Details

    Description

      seekToBeginning and seekToEnd perform 2 operations:

      1. retrieve assigned partitions from the subscription state if no partitions provided by the user
      2. request offsets reset for those partitions (to EARLIEST or LATEST offset) (update partition state in the subscription state)

      In the new asyn consumer, these operation are performed sequentially in the app thread, which could lead to errors if retrieving the assigned partitions first, and then requesting the reset: the assignment could change in the background thread in-between 1) and 2) and the reset would fail with
      "No current assignment for partition"
       

      We should move the implementation to the background thread, where all modifications to the subscription state should occur. We could introduce a new event (something like OffsetResetEvent or similar) that would take as input the set of partitions received in the api calls and the OffsetResetStrategy (EARLIEST or LATEST). When processing it in the background, it would safely perform the same 2 actions that are currently in the app thread seekToBeginning/End.

      Attachments

        Issue Links

          Activity

            People

              taijuwu TaiJuWu
              lianetm Lianet Magrans
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: