Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.15.0
-
None
Description
Dataset Lineage emission in current Gobblin code is only available for batch mode of execution. For instance, the lineage events are emitted on dataset commit, which is never invoked for long-running tasks. This PR refactors lineage related code so that it is leverageable in streaming mode of execution as well.