Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
2.4.0
-
None
-
None
Description
Let's assume we have a writer data source which writes to a dynamic schema (e.g. writing to a database table). If we have mode append but the table is missing then writing will fail.
The reason for this is that in the dataframeWriter save method we have val relation = DataSourceV2Relation.create(source, options) which in turn contains: val reader = source.createReader(options, userSpecifiedSchema)
The problem is that this sets the schema from the reader portion of the data source. If the data source can't infer the schema then we will fail to write (even thought the schema is actually part of the dataframe).
Attachments
Issue Links
- duplicates
-
SPARK-25700 Avoid to create a readsupport at write path in Data Source V2
- Resolved