Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.0
-
None
-
None
Description
To validate writes with DataSourceV2, the planner needs to get a source's schema. The current API has no direct way to get that schema. SPARK-23321 instantiates a reader to get the schema, but sources are not required to implement ReadSupport or ReadSupportWithSchema. V2 should either add a method to get the schema of a source, or require sources implement ReadSupport.
Attachments
Issue Links
- is superceded by
-
SPARK-25528 data source V2 API refactoring (batch read)
- Resolved