Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5267

Add a streaming module to ingest Apache Camel Messages from a configured endpoints

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Done
    • 1.2.0
    • None
    • DStreams

    Description

      The number of input stream protocols supported by Spark Streaming is quite limited, which constrains the number of systems with which it can be integrated.

      This proposal solves the problem by adding an optional module that integrates Apache Camel, which supports many additional input protocols. Our tried and tested implementation of this proposal is "spark-streaming-camel".

      An Apache Camel service is run on a separate Thread, consuming each http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Message.html and storing it into Spark's memory. The provider of the Message is specified by any consuming component URI documented at http://camel.apache.org/components.html, making all of these protocols available to Spark Streaming.

      Thoughts?

      Attachments

        Activity

          People

            Unassigned Unassigned
            sbrewin@apache.org Steve Brewin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 120h
                120h
                Remaining:
                Remaining Estimate - 120h
                120h
                Logged:
                Time Spent - Not Specified
                Not Specified