Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-16914

camel-kafka: possible corruption of idempotency messages when using KafkaIdempotentRepository

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.11.1
    • 3.11.3, 3.12.0
    • camel-kafka
    • None
    • Unknown

    Description

      FakeDataProducer.javaMessageLossReproducer.javaI noticed that when using the KafkaIdempotentRepository, sometimes it seems that it is losing or corrupting the ID information, causing loss of messages.

       

      When reading the topic used for publishing the idempotency data with kafkacat I get:

       

      ERROR: Failed to format message in fake-data-inserts [1] at offset 39: expected key end-of-input, but 2 bytes remaining : terminating

       

      As a result, this causes the loss of messages because the consumers cannot go further than the corrupted record.

       

      I found this while using Camel K and Strimzi (on a setup with 3 Kafkas using 1 topic with 3 partitions).

       

      I couldn't test w/ 3.12 yet, but I suspect it would also have the same problem.

       

       

      Attachments

        1. FakeDataProducer.java
          2 kB
          Otavio Rodolfo Piske
        2. MessageLossReproducer.java
          4 kB
          Otavio Rodolfo Piske

        Activity

          People

            orpiske Otavio Rodolfo Piske
            orpiske Otavio Rodolfo Piske
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: