Uploaded image for project: 'Bahir (Retired)'
  1. Bahir (Retired)
  2. BAHIR-167 Update Bahir to use the new Spark 2.3.0 and datasource v2 API.
  3. BAHIR-164

Spark Streaming with MQTT fails with Spark 2.3.0

    XMLWordPrintableJSON

Details

    Description

      Currently I’m not able to use latest Spark 2.3.0 release together with Bahir, but getting the following error:
       
      java.lang.AssertionError: assertion failed: DataFrame returned by getBatch from org.apache.bahir.sql.streaming.mqtt.MQTTTextStreamSource@66847d45 did not have isStreaming=true
       
      org/apache/bahir/sql/streaming/mqtt/MQTTStreamSource.scala line 164 seems to be causing the issue, i.e. it’s returning a normal dataframe instead of streaming dataframe.
       
      There seems to be discussion about the same topic, but on stream source for Kafka: https://lists.apache.org/thread.html/015f2fbe9fbeafb86214a969a2d883a4898621189a86f21a4c6fbb75@%3Cdev.spark.apache.org%3E

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #65

          Activity

            People

              prashant Prashant Sharma
              jussipekkap Jussi-Pekka Partanen
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: