Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36533

Allow streaming queries with Trigger.Once run in multiple batches

    XMLWordPrintableJSON

Details

    Description

      Currently streaming queries with Trigger.Once will always load all of the available data in a single batch. Because of this, the amount of data the queries can process is limited, or Spark driver will be out of memory.

      We should allow streaming queries with Trigger.Once run in multiple batches.

      Attachments

        Activity

          People

            bozhang Bo Zhang
            bozhang Bo Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: