Create a Bronze Iceberg table with a permissive schema (all columns as strings or variant) and ingest raw source data via a streaming or batch job, preserving the original payload without transformation
Create a Silver Iceberg table with an enforced schema (typed columns, not_null constraints where applicable) and run a transformation job that reads Bronze, validates and casts types, and writes to Silver using MERGE or INSERT OVERWRITE
Create a Gold Iceberg table containing business-level aggregates or dimensional models, populated by transformations reading from Silver
Use Iceberg schema evolution (add_column, rename_column) to handle upstream schema changes in Bronze without rewriting Silver immediately; adapt Silver transformation logic to handle old and new schemas using conditional column expressions
Implement snapshot-based auditing by reading Silver with a specific snapshot ID in data quality validation jobs before promoting new Silver data to Gold
Known gotchas
Writing raw data to Bronze as strings avoids ingestion failures but defers type errors to the Silver layer; Bronze-to-Silver validation failures require a dead-letter path and alerting to avoid silently dropping records
Iceberg schema enforcement is at the catalog level, not enforced by the storage format; a misconfigured writer can still write files with a different schema that the catalog rejects at commit time, causing partial write failures
Running MERGE or INSERT OVERWRITE on large Silver tables for every Bronze batch can be expensive; consider micro-batch incremental strategies (streaming MERGE or position-based partition overwrite) to limit the rewrite scope per batch
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp