Design a BigQuery materialized view that qualifies for incremental refresh and automatic query rewrite

domain: data-engineering · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Keep the defining query within incremental-MV limits: no LEFT/RIGHT/FULL OUTER JOIN, no window functions like ROW_NUMBER(), and aggregate outputs can't be further filtered/computed on or used as clustering columns.
  2. If a JOIN is required, ensure only the left-side table(s) receive appended data — incremental MVs can't incrementally absorb append-only changes on the right side of a join.
  3. Accept that OUTER JOIN/HAVING/UNION/analytic functions are allowed but make the view non-incremental — losing both incremental refresh and automatic query rewrite (smart tuning).
  4. Set max_staleness deliberately: if the view goes more than 3 days without a refresh, queries against it start failing rather than silently returning stale data.
  5. Stay under per-base-table caps — 100 materialized views per base table per project, 500 per base table per org — when planning multiple MVs over the same source.
  6. Verify smart tuning is engaging by checking the query plan for a READ <materialized_view> step when querying the base table directly; it doesn't engage over tables with active CDC.

Known gotchas

Related routes

Create a BigQuery materialized view with automatic incremental refresh for a frequently queried aggregation
cloud.google.com · 5 steps · unrated
Create a TimescaleDB continuous aggregate with an incremental refresh policy and enable real-time aggregation for sub-interval queries
docs.timescale.com · 6 steps · unrated
Configure Snowflake dynamic tables with incremental and full refresh modes for automated pipeline materialization
docs.snowflake.com · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans