Description
https://github.com/apache/spark/commit/5fef6e3513d6023a837c427d183006d153c7102b introduced append mode support in Datasource V2.
The problem is, write path ended up
Retrieving the physical schema of the underlying storage is potentially expensive. Actually even worse: it looks odd that write path requires read side's schema.
This is a regression comparing to Spark 2.3.
Attachments
Issue Links
- is duplicated by
-
SPARK-25709 Can't append to a data source V2 unless target already has schema information or fixed schema matching dataframe
- Resolved
- links to
(1 links to)