Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
0.9.0.1
-
None
-
None
-
kafka: kafka_2.11-0.9.0.1
java: jdk1.8.0_65
OS: Linux stephen-T450s 3.19.0-51-generic #57~14.04.1-Ubuntu SMP Fri Feb 19 14:36:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Description
Overview
=======
The results of test are not consistent.
The problem is that something is preventing the consumer from receiving the messages.
Configuration
==========
Server (only num.partitions is changed)
diff config/server.properties config.backup/server.properties
65c65
< num.partitions=8
—
> num.partitions=1
Producer
properties.put("bootstrap.servers", “localhost:9092”);
properties.put("acks", "all");
properties.put("key.serializer", "org.apache.kafka.common.serialization.LongSerializer");
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
properties.put("partitioner.class", "kafkatest.sample2.SimplePartitioner");
Consumer
properties.put("bootstrap.servers", “localhost:9092”);
properties.put("group.id", "testGroup");
properties.put("key.deserializer", "org.apache.kafka.common.serialization.LongDeserializer");
properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
properties.put("enable.auto.commit", "false");
Steps to reproduce:
===============
1. started the zookeeper
2. started the kafka server
3. created topic
$ bin/kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partition 8 --topic testTopic4
4. Ran SimpleProducerDriver with 5 producers, and the amount of messages produced is 50
5. Offset Status
$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic testTopic4 --time -1
testTopic4:2:1
testTopic4:5:27
testTopic4:4:1
testTopic4:7:2
testTopic4:1:8
testTopic4:3:0
testTopic4:6:11
testTopic4:0:0
6. waited till the producer driver completes, it takes no more than a few seconds
7. ran the SimpleConsumerDriver a couple of times, and no message is received. Following DEBUG information is found:
2016-02-25 22:42:19 DEBUG [pool-1-thread-2] Fetcher: - Ignoring fetched records for partition testTopic4-3 since it is no longer fetchable
8. altered the properties of consumer, had the auto commit disabled:
//properties.put("enable.auto.commit", "false");
9. ran the SimpleConsumerDriver a couple of times, still, no message is received.
Following DEBUG information is found:
2016-02-25 22:47:23 DEBUG [pool-1-thread-2] ConsumerCoordinator: - Committed offset 8 for partition testTopic4-1
seems like the offset was updated?
10. re-enabled the auto commit, nothing changed.
Following DEBUG information is found:
2016-02-25 22:49:38 DEBUG [pool-1-thread-7] Fetcher: - Resetting offset for partition testTopic4-6 to the committed offset 11
11. ran the SimpleProducerDriver again, another 50 messages are published
12. ran the SimpleConsumerDriver again, 100 messages were consumed.
13. ran the SimpleConsumerDriver again, 50 messages were consumed.
As auto commit is disabled, all messages (100) should be consumed.
The results of test are not consistent.
The problem is that something is preventing the consumer from receiving the messages.
And sometimes it required running the producer when the consumers are still active so as to get around it.
And once the consumers started to consume messages, the problem did not occur any more.