Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3293

Consumers are not able to get messages.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 0.9.0.1
    • None
    • consumer, offset manager
    • None
    • kafka: kafka_2.11-0.9.0.1
      java: jdk1.8.0_65
      OS: Linux stephen-T450s 3.19.0-51-generic #57~14.04.1-Ubuntu SMP Fri Feb 19 14:36:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

    Description

      Overview
      =======
      The results of test are not consistent.
      The problem is that something is preventing the consumer from receiving the messages.

      Configuration
      ==========
      Server (only num.partitions is changed)
      diff config/server.properties config.backup/server.properties
      65c65
      < num.partitions=8

      > num.partitions=1

      Producer
      properties.put("bootstrap.servers", “localhost:9092”);
      properties.put("acks", "all");
      properties.put("key.serializer", "org.apache.kafka.common.serialization.LongSerializer");
      properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
      properties.put("partitioner.class", "kafkatest.sample2.SimplePartitioner");

      Consumer
      properties.put("bootstrap.servers", “localhost:9092”);
      properties.put("group.id", "testGroup");
      properties.put("key.deserializer", "org.apache.kafka.common.serialization.LongDeserializer");
      properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
      properties.put("enable.auto.commit", "false");

      Steps to reproduce:
      ===============
      1. started the zookeeper
      2. started the kafka server
      3. created topic
      $ bin/kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partition 8 --topic testTopic4
      4. Ran SimpleProducerDriver with 5 producers, and the amount of messages produced is 50
      5. Offset Status
      $ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic testTopic4 --time -1
      testTopic4:2:1
      testTopic4:5:27
      testTopic4:4:1
      testTopic4:7:2
      testTopic4:1:8
      testTopic4:3:0
      testTopic4:6:11
      testTopic4:0:0
      6. waited till the producer driver completes, it takes no more than a few seconds
      7. ran the SimpleConsumerDriver a couple of times, and no message is received. Following DEBUG information is found:
      2016-02-25 22:42:19 DEBUG [pool-1-thread-2] Fetcher: - Ignoring fetched records for partition testTopic4-3 since it is no longer fetchable
      8. altered the properties of consumer, had the auto commit disabled:
      //properties.put("enable.auto.commit", "false");
      9. ran the SimpleConsumerDriver a couple of times, still, no message is received.
      Following DEBUG information is found:
      2016-02-25 22:47:23 DEBUG [pool-1-thread-2] ConsumerCoordinator: - Committed offset 8 for partition testTopic4-1
      seems like the offset was updated?
      10. re-enabled the auto commit, nothing changed.
      Following DEBUG information is found:
      2016-02-25 22:49:38 DEBUG [pool-1-thread-7] Fetcher: - Resetting offset for partition testTopic4-6 to the committed offset 11
      11. ran the SimpleProducerDriver again, another 50 messages are published
      12. ran the SimpleConsumerDriver again, 100 messages were consumed.
      13. ran the SimpleConsumerDriver again, 50 messages were consumed.
      As auto commit is disabled, all messages (100) should be consumed.

      The results of test are not consistent.
      The problem is that something is preventing the consumer from receiving the messages.
      And sometimes it required running the producer when the consumers are still active so as to get around it.
      And once the consumers started to consume messages, the problem did not occur any more.

      Attachments

        Activity

          People

            nehanarkhede Neha Narkhede
            stephen.wong Stephen Wong
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: