Avro schema typedef and support for Avro schema evolution in Atlas

Currently the base types in Atlas do not include Avro schemas. It would be nice to add typedef for Avro schema and any associated metadata to support schema evolution.

• For example, Avro_schema type supports:
  • All avro types, both primitive and complex, including union types, as fields of schema
  • All types have doc strings and defaults
  • A field of a schema can be another schema
  • Indefinite nesting of records, arrays.
  • Associated entities array attribute contains pointers to all datasets that reflect the avro schema
  • Fully expanded avroNotation for use in serDe
  • Schema evolution features such as isLatest (Boolean) and version number
• Schema evolution Process
  • Input: avro schema
  • Output: new version of avro schema
  • Compatibility: FULL, BACKWARD, FORWARD, NONE
  • IsBreakingChange (Boolean): does the change produce an incompatible schema? (ie its compatibility is not “FULL”)
    *