Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34378

Loosen AvroSerializer validation to allow extra nullable user-provided fields

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.1
    • 3.3.0
    • SQL
    • None

    Description

      Currently, when writing out Avro data using a custom schema (avroSchema), if there are any extra Avro fields which do not have a matching field in the Catalyst schema, the serialization will fail. This is much more strict than on the deserialization path, where Avro fields not present in the Catalyst schema are ignored, and Catalyst fields not present in the Avro schema are allowed as long as they are nullable. I believe it will be more user-friendly if extra Avro fields are allowed, as long as they are optional. This makes it easier for users to write out data with Avro schemas which may be outside of their control.

      If there is concern about the safety of this approach (i.e. there are use cases where users want strict matching), we can make it configurable.

      Attachments

        Issue Links

          Activity

            People

              xkrogen Erik Krogen
              xkrogen Erik Krogen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: