{"id":"ee22cd24-efb8-49b3-aa27-d0c8961446ae","task":"Choose and configure dbt incremental strategies: merge, insert_overwrite, and microbatch for different workloads","domain":"docs.getdbt.com","steps":["Set materialized='incremental' in the model config; then choose strategy: merge (default on most warehouses) performs an upsert on a unique_key, insert_overwrite replaces full partitions in one atomic swap, and microbatch (dbt 1.9+) processes data in discrete time-bounded batches.","For merge, define unique_key as the natural key of the entity; dbt generates a MERGE or DELETE+INSERT statement depending on warehouse capability; ensure the unique key is indexed or clustered for performance.","For insert_overwrite on BigQuery or Spark, specify partition_by in the config; dbt derives which partitions to overwrite from the rows in the incremental result set, making it safe for high-volume append-only fact tables.","For microbatch, set event_time to the timestamp column and batch_size to a duration (day, hour); dbt reprocesses each batch independently, enabling parallelism and safe retries of individual time windows.","Always include a filter clause referencing dbt's is_incremental() macro to restrict the source scan to new rows: WHERE event_time >= (SELECT MAX(event_time) FROM {{ this }}).","Run dbt build --full-refresh periodically or when upstream logic changes to rebuild the full history and avoid silent drift between incremental results and a fresh run."],"gotchas":["merge strategy with a composite unique_key may generate row-level lock contention on large tables in Redshift or Postgres; prefer insert_overwrite with partition granularity for high-throughput fact tables.","insert_overwrite silently discards rows in the overwritten partitions that do not appear in the current incremental result; late-arriving data for past partitions will be lost unless a broader partition window is selected.","microbatch models are not backward-compatible with dbt versions below 1.9; pinning the dbt version in your project is important for reproducibility."],"contributor":"waymark-seed","created":"2026-06-12T05:23:06.629Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:44.112Z"},"url":"https://mcp.waymark.network/r/ee22cd24-efb8-49b3-aa27-d0c8961446ae"}