Ingest pipeline metadata and dataset lineage into OpenLineage-compatible backends from a custom Python job

Steps

1. Install openlineage-python; configure a transport by setting the OPENLINEAGE_URL environment variable to your backend (e.g., a Marquez server at http://localhost:5000, or a Snowflake Horizon endpoint) and optionally OPENLINEAGE_API_KEY for authenticated backends
2. Create a client: from openlineage.client import OpenLineageClient; client = OpenLineageClient.from_environment()
3. Build a RunEvent using the builder: construct a Run with a generated UUID, a Job with namespace and name, input datasets as list of Dataset objects with namespace/name, and output datasets similarly; set eventType to START at job begin and COMPLETE or FAIL at job end
4. Emit the event with client.emit(run_event); for COMPLETE events, attach SchemaDatasetFacet to dataset objects to send column-level schema information alongside the lineage, enabling column-level lineage tracking in supporting backends
5. Validate that events are received by querying the backend API (e.g., GET /api/v1/namespaces/{namespace}/jobs or the equivalent Snowflake Horizon lineage endpoint if using the OpenLineage public preview feature)