Details
-
Task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Previously in streaming pipeline, to avoid race condition on metadata schema, when we do hive registration, we always fetch the latest schema from Kafka SchemaRegistry, since gobblin converter may change the schema, this introduce discrepancy between hive table schema and real file schema, so we need a better way to solve this problem