Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.2.0
Description
Currently streaming queries with Trigger.Once will always load all of the available data in a single batch. Because of this, the amount of data the queries can process is limited, or Spark driver will be out of memory.
We should allow streaming queries with Trigger.Once run in multiple batches.