Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6106

Postpone normal processing of tasks within a thread until restoration of all tasks have completed

    XMLWordPrintableJSON

Details

    Description

      Let's say a stream thread hosts multiple tasks, A and B. At the very beginning when A and B are assigned to the thread, the thread state is TASKS_ASSIGNED, and the thread start restoring these two tasks during this state using the restore consumer while using normal consumer for heartbeating.

      If task A's restoration has completed earlier than task B, then the thread will start processing A immediately even when it is still in the TASKS_ASSIGNED phase. But processing task A will slow down restoration of task B since it is single-thread. So the thread's transition to RUNNING when all of its assigned tasks have completed restoring and now can be processed will be delayed.

      Note that the streams instance's state will only transit to RUNNING when all of its threads have transit to RUNNING, so the instance's transition will also be delayed by this scenario.

      We'd better to not start processing ready tasks immediately, but instead focus on restoration during the TASKS_ASSIGNED state to shorten the overall time of the instance's state transition.

      Attachments

        Activity

          People

            ckamal Kamal Chandraprakash
            guozhang Guozhang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: