Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
This issue covers any API changes, refactoring, and utility classes/methods that are necessary to make it possible to implement support for Kafka 0.10 sources and sinks in Structured Streaming.
At a minimum, the changes in SPARK-16963 will be needed in order for the Kafka commit protocol to work. Given that KIP-33 ("Add a time based log index") is not yet in place, it may be necessary to make additional API changes in Spark for commit to work efficiently. Some refactoring of the existing KafkaRDD class is probably in order also.