Redesign Kafka Component Integration

NiFi has supported integration with Kafka across a number of major release versions. As Kafka has continued to release new major versions, NiFi integration has followed a general pattern of duplicating existing code, updating Kafka libraries, and making small modifications. Although this approach has allowed NiFi to support a range of Kafka versions, it has made Kafka components difficult to maintain.

A new approach should be designed and implemented that abstracts access to Kafka library components. This approach should provide more maintainable Kafka Processors, and also define a better path to support new major Kafka releases.

The general approach should consist of a NiFi Controller Service and supporting classes that abstract Kafka operations for publishing and consuming messages. The Controller Service interface should not have any dependencies on Kafka libraries, making it independent of Kafka versions. The interface should define a contract for accessing Kafka Brokers so that Controller Service implementations will be responsible for connection and authentication settings. Controller Service implementations should be aligned to major Kafka versions, allowing minor dependency version updates without changing Controller Service implementations.

New Kafka Processors should be designed and implemented to use the Controller Service interface, without requiring knowledge of Kafka connection details. New Processors should support the same basic production and consumption capabilities, with configurable strategies for Kafka Record handling. These Processors should support the same capabilities as current Processors, including treating entire FlowFiles as Kafka Records, as well as processing FlowFiles using a configurable delimiter or using NiFi Record services.

New Kafka components should build on the pattern of the nifi-kafka-shared module to promote code reuse across major Kafka versions.

The new decoupled Kafka Processors and Controller Services should enable better unit testing. The Testcontainers library should considered for integration testing with different Kafka runtime versions.

Documentation should be written to describe common migration scenarios from existing Kafka Processors to new components.