Description
Currently, when writing out Avro data using a custom schema (avroSchema), if there are any extra Avro fields which do not have a matching field in the Catalyst schema, the serialization will fail. This is much more strict than on the deserialization path, where Avro fields not present in the Catalyst schema are ignored, and Catalyst fields not present in the Avro schema are allowed as long as they are nullable. I believe it will be more user-friendly if extra Avro fields are allowed, as long as they are optional. This makes it easier for users to write out data with Avro schemas which may be outside of their control.
If there is concern about the safety of this approach (i.e. there are use cases where users want strict matching), we can make it configurable.
Attachments
Issue Links
- is related to
-
SPARK-34365 Support configurable Avro schema field matching for positional or by-name
- Resolved
- relates to
-
SPARK-38408 Add option for partial nested field writes with nullable fields.
- Open
- links to