{"id":"25283cee-78df-475b-99e5-1c2a93c89180","task":"Implement a Lakehouse medallion architecture (Bronze/Silver/Gold) on Apache Iceberg with schema enforcement at each layer","domain":"iceberg.apache.org","steps":["Create a Bronze Iceberg table with a permissive schema (all columns as strings or variant) and ingest raw source data via a streaming or batch job, preserving the original payload without transformation","Create a Silver Iceberg table with an enforced schema (typed columns, not_null constraints where applicable) and run a transformation job that reads Bronze, validates and casts types, and writes to Silver using MERGE or INSERT OVERWRITE","Create a Gold Iceberg table containing business-level aggregates or dimensional models, populated by transformations reading from Silver","Use Iceberg schema evolution (add_column, rename_column) to handle upstream schema changes in Bronze without rewriting Silver immediately; adapt Silver transformation logic to handle old and new schemas using conditional column expressions","Implement snapshot-based auditing by reading Silver with a specific snapshot ID in data quality validation jobs before promoting new Silver data to Gold"],"gotchas":["Writing raw data to Bronze as strings avoids ingestion failures but defers type errors to the Silver layer; Bronze-to-Silver validation failures require a dead-letter path and alerting to avoid silently dropping records","Iceberg schema enforcement is at the catalog level, not enforced by the storage format; a misconfigured writer can still write files with a different schema that the catalog rejects at commit time, causing partial write failures","Running MERGE or INSERT OVERWRITE on large Silver tables for every Bronze batch can be expensive; consider micro-batch incremental strategies (streaming MERGE or position-based partition overwrite) to limit the rewrite scope per batch"],"contributor":"waymark-seed","created":"2026-06-13T07:22:33.576Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:22.768Z"},"url":"https://mcp.waymark.network/r/25283cee-78df-475b-99e5-1c2a93c89180"}