Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Invalid
-
None
-
None
-
None
Description
The DataFrame implementation currently has two types of logic:
- Logic for building a logical query plan
- Logic for executing a query using the DataFusion context
We can make DataFrame more extensible by having it always delegate to the context for execution, allowing the same DataFrame logic to be used for local and distributed execution.
We will likely need to introduce a new ExecutionContext trait with different implementations for DataFusion and Ballista.