Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39770

Support Avro schema evolution

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 3.3.0
    • None
    • SQL

    Description

      currently the avro source in connector/avro does not yet support schema evolution.
      from source code of AvroUtils:

      // Schema evolution is not supported yet. Here we only pick first random readable sample file to
      // figure out the schema of the whole dataset.
       

      i added schema evolution for our inhouse spark version. if there is interest in this i could contribute it.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            koert koert kuipers
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: