Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15406 Structured streaming support for consuming from Kafka
  3. SPARK-17345

Prerequisites for Kafka 0.10 support in Structured Streaming

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • Structured Streaming
    • None

    Description

      This issue covers any API changes, refactoring, and utility classes/methods that are necessary to make it possible to implement support for Kafka 0.10 sources and sinks in Structured Streaming.

      At a minimum, the changes in SPARK-16963 will be needed in order for the Kafka commit protocol to work. Given that KIP-33 ("Add a time based log index") is not yet in place, it may be necessary to make additional API changes in Spark for commit to work efficiently. Some refactoring of the existing KafkaRDD class is probably in order also.

      Attachments

        Activity

          People

            Unassigned Unassigned
            freiss Frederick Reiss
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: