Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9941

WorkerSinkTask:When a record triggers a RetriableException and the retry is processed successfully, its offset does not commit.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.2.0
    • 2.2.0
    • connect
    • None

    Description

      When a record triggers a RetriableException and the retry is processed successfully, its offset does not commit.

      The processing process of connect is as follows:

      1: commitOffsets

      2: poll (long timeoutMs)

      3: convertMessages

      4: deliverMessages

      Offset storage:

      1: lastCommittedOffsets

      2: currentOffsets

      3: origOffsets

      Cause of the problem: I need to retry the record

      1.RetriableException from SinkTask:

      2.Pausing partitions

      3.Not returning fetched records for assigned partition since it is no longer fetchable

         3.1.ConsumerRecords<byte[], byte[]> msgs = pollConsumer(timeoutMs);

         3.2.convertMessages(msgs);// msgs is empty

         3.3.origOffsets.clear(); // Record of the retry operation, it's offset has been cleared

      4.The retry operation has completed

          4.1.now commit the offset of the  record of the retry operation, but origOffsets has been cleared.

          4.2.Skipping offset commit, no change since last commit

          4.3.Finished offset commit successfully in 0 ms for sequence number 384: null

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jiangchuan-java jiangchuan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: