Implement a Lakehouse medallion architecture (Bronze/Silver/Gold) on Apache Iceberg with schema enforcement at each layer

Steps

1. Create a Bronze Iceberg table with a permissive schema (all columns as strings or variant) and ingest raw source data via a streaming or batch job, preserving the original payload without transformation
2. Create a Silver Iceberg table with an enforced schema (typed columns, not_null constraints where applicable) and run a transformation job that reads Bronze, validates and casts types, and writes to Silver using MERGE or INSERT OVERWRITE
3. Create a Gold Iceberg table containing business-level aggregates or dimensional models, populated by transformations reading from Silver
4. Use Iceberg schema evolution (add_column, rename_column) to handle upstream schema changes in Bronze without rewriting Silver immediately; adapt Silver transformation logic to handle old and new schemas using conditional column expressions
5. Implement snapshot-based auditing by reading Silver with a specific snapshot ID in data quality validation jobs before promoting new Silver data to Gold