Description
For a large number of topics and partition the broker can encounter this issue:
[2022-09-12 14:14:42,114] ERROR [BrokerMetadataSnapshotter id=4] Unexpected error handling CreateSnapshotEvent (kafka.server.metadata.BrokerMetadataSnapshotter) org.apache.kafka.raft.errors.BufferAllocationException: Append failed because we failed to allocate memory to write the batch at org.apache.kafka.raft.internals.BatchAccumulator.append(BatchAccumulator.java:161) at org.apache.kafka.raft.internals.BatchAccumulator.append(BatchAccumulator.java:112) at org.apache.kafka.snapshot.RecordsSnapshotWriter.append(RecordsSnapshotWriter.java:167) at kafka.server.metadata.RecordListConsumer.accept(BrokerMetadataSnapshotter.scala:49) at kafka.server.metadata.RecordListConsumer.accept(BrokerMetadataSnapshotter.scala:42) at org.apache.kafka.image.TopicImage.write(TopicImage.java:78) at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:79) at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:129) at kafka.server.metadata.BrokerMetadataSnapshotter$CreateSnapshotEvent.run(BrokerMetadataSnapshotter.scala:116) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173) at java.base/java.lang.Thread.run(Thread.java:829)
This can happen because the snapshot is larger than 5 * 8MB.
Attachments
Issue Links
- relates to
-
KAFKA-10652 Raft leader should flush accumulated writes after a min size is reached
- Resolved
- links to