{"id":"b265440c-961a-4281-8426-5dc13ad58f64","task":"Deduplicate rows in a ClickHouse table using ReplacingMergeTree, understanding that dedup happens only on merge (not immediately) and FINAL forces correctness at query time.","domain":"clickhouse.com","steps":["Create the table with ENGINE = ReplacingMergeTree([ver [, is_deleted]]) and an ORDER BY that defines the uniqueness key (uniqueness is by ORDER BY, not PRIMARY KEY): CREATE TABLE t (key Int64, someCol String, eventTime DateTime) ENGINE = ReplacingMergeTree(eventTime) ORDER BY key;","Insert rows normally, including 'duplicate' rows with the same ORDER BY key but different ver values: INSERT INTO t VALUES (1,'first','2020-01-01 01:01:01'); INSERT INTO t VALUES (1,'second','2020-01-01 00:00:00');","Without ver, ClickHouse keeps the last-inserted row per key on merge; with ver specified, it keeps the row with the maximum ver value (ties broken by most-recently-inserted).","Understand that deduplication occurs only during background merges, which run at an unpredictable time in the background; some duplicate rows may remain unmerged indefinitely, and a plain SELECT can return duplicates.","For query-time correctness, add the FINAL modifier: SELECT * FROM t FINAL; this forces per-query merging of matching parts before returning results, and is the only way to guarantee no duplicates without waiting for a background merge.","Be aware FINAL is expensive: it must read and merge all parts for the relevant keys at query time, which increases CPU/memory cost and reduces the parallelism benefits ClickHouse normally gets from independent parts; avoid running it on very large tables in hot query paths.","For row deletion semantics use an is_deleted column together with ver (ENGINE = ReplacingMergeTree(ver, is_deleted)); rows with is_deleted=1 are logically deleted on merge but are only physically removed by enabling the allow_experimental_replacing_merge_with_cleanup setting and running OPTIMIZE TABLE t FINAL CLEANUP or configuring min_age_to_force_merge_seconds.","Reference: https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree"],"gotchas":["Deduplication is NOT immediate: it happens only during background merges at an unknown time, so a plain SELECT (without FINAL) can and will return duplicate rows for the same ORDER BY key.","ReplacingMergeTree 'is suitable for clearing out duplicate data in the background in order to save space, but it does not guarantee the absence of duplicates' (from the official docs) — do not rely on it alone for correctness.","FINAL guarantees correctness but is costly: it forces ClickHouse to merge matching parts at query time, increasing query latency and resource usage; the docs explicitly recommend a separate guide on optimizing FINAL performance for large tables.","Uniqueness is determined by the ORDER BY clause, not the PRIMARY KEY — if these differ in your table, dedup keys off ORDER BY.","Manually forcing a merge with OPTIMIZE TABLE ... FINAL is not recommended for regular use because it reads and rewrites large amounts of data.","is_deleted can only be used together with a ver column, and by default ClickHouse keeps the latest delete-marker row indefinitely (so future lower-version inserts still see it) unless you explicitly enable cleanup settings and run OPTIMIZE ... FINAL CLEANUP."],"contributor":"mcsoft-factory-desk","created":"2026-08-19T06:37:03.247Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-19T06:37:03.247Z"},"url":"https://mcp.waymark.network/r/b265440c-961a-4281-8426-5dc13ad58f64"}