{"id":"0fa3d5f6-8558-43d5-a720-998820e2b2ca","task":"Enforce data quality in a Lakeflow Declarative Pipeline (formerly Delta Live Tables) using expectations","domain":"data-engineering","steps":["Decorate a Python table/view function with @dp.expect(\"name\", \"constraint_sql\") to log violations while still writing the rows — this is the default warn-and-keep behavior.","Use @dp.expect_or_drop(\"name\", \"constraint_sql\") to silently drop violating rows before they're written to the target table.","Use @dp.expect_or_fail(\"name\", \"constraint_sql\") to stop pipeline execution immediately on the first violation.","For multiple constraints at once, use the grouped forms expect_all / expect_all_or_drop / expect_all_or_fail, passing a dict of {name: constraint}.","In SQL pipelines, use CONSTRAINT <name> EXPECT (<expr>) ON VIOLATION {FAIL UPDATE | DROP ROW} inline on CREATE STREAMING TABLE / CREATE MATERIALIZED VIEW — omitting ON VIOLATION defaults to warn-and-keep.","Check the pipeline event log's ExpectationMetrics for per-constraint pass/fail row counts rather than inferring quality from before/after row counts."],"gotchas":["The default behavior (plain @dp.expect / no ON VIOLATION clause) keeps failing rows in the output table — teams that assume \"expectation\" implies filtering are often surprised bad rows are still written.","expect_or_fail halts the entire pipeline run on the first violation, which is a blunt instrument for high-volume streaming sources — a single bad record can stop an otherwise-healthy pipeline."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/0fa3d5f6-8558-43d5-a720-998820e2b2ca"}