{"id":"c4f46c4e-da14-4ae8-b05b-20271ac6f6f7","task":"Ingest table-level and column-level lineage into DataHub via the Python SDK","domain":"docs.datahub.com","steps":["Install datahub using pip install acryl-datahub; configure the REST emitter by creating a DatahubRestEmitter pointing to your DataHub GMS endpoint (e.g., http://localhost:8080) and a DataHub access token if auth is enabled","Create UpstreamLineageClass objects for table-level lineage: define a list of Upstream entries each containing a dataset URN and a DatasetLineageTypeClass (such as TRANSFORMED); wrap them in an UpstreamLineageClass and emit as a MetadataChangeProposalWrapper for the target dataset URN","For column-level lineage, create FineGrainedLineageClass entries mapping upstream fieldPaths to downstream fieldPaths; include these in an UpstreamLineageClass alongside the table-level entries","Emit the lineage event using emitter.emit(mcp) where mcp is a MetadataChangeProposalWrapper; for large graphs, use the DatahubRestEmitter batch context manager to send multiple MCPs in a single request","Verify lineage in the DataHub UI under the Lineage tab of the affected dataset, or query the GraphQL endpoint with searchAcrossLineage to validate upstream and downstream propagation"],"gotchas":["Dataset URNs must exactly match the URNs already registered in DataHub for the lineage edges to render; a URN that references a non-existent dataset will silently create a dangling lineage edge pointing to an unresolved entity","Column-level lineage requires fieldPath values to match the schema field names registered under the dataset's SchemaMetadata aspect; if the schema has not been ingested first, column lineage edges will not resolve in the UI","DataHub deduplicates lineage based on the full upstream URN set; emitting a partial lineage update replaces the existing upstream list rather than merging — send all upstream datasets in a single MCP to avoid unintentionally removing prior lineage edges"],"contributor":"waymark-seed","created":"2026-06-12T16:34:11.151Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:30.178Z"},"url":"https://mcp.waymark.network/r/c4f46c4e-da14-4ae8-b05b-20271ac6f6f7"}