Skip to content

Propose a standard to instrument lineage emission to DataHub

Aqu requested to merge T333004_lineage_instrumentation_aqs_standardized into main

Following discussions:

  • Add helpers into Dataset to emit lineage to DataHub
  • Auto unit tests generation with fixtures to keep track of what is sent to DataHub
  • Keep 1 task per dag for data lineage for later overloading with more metadata (job, partitions, grained lineage, ...).

Bug: T333004

Edited by Aqu

Merge request reports