{"id":"9c307ed8-542a-4f8f-ac43-3451631e2d2e","task":"Use Debezium to capture MySQL or PostgreSQL change events and stream them to Kafka","domain":"dataeng-general","steps":["Enable the database's binary log (MySQL: binlog_format=ROW; PostgreSQL: wal_level=logical and create a replication slot) and grant Debezium a user with replication privileges","Deploy the Debezium source connector via Kafka Connect REST API with the appropriate connector class and database connection properties","Verify that the connector performs an initial snapshot of existing table data by reading the snapshot.mode setting (default: initial)","Monitor the Kafka topic for the captured table (named by default as server.schema.table) and confirm that insert, update, and delete events appear with the correct before and after payloads","Configure tombstone events and topic cleanup policy (compact) on the Kafka topic to support downstream consumers that rely on log compaction for key-based deduplication"],"gotchas":["PostgreSQL replication slots accumulate WAL segments until the slot is consumed; a stalled or dropped Debezium connector that does not consume the slot will cause unbounded WAL growth and can fill the disk","MySQL GTID mode and non-GTID mode require different snapshot and failover configurations in Debezium; mixing them or switching modes after connector creation can cause the connector to lose its position in the binlog","Debezium's initial snapshot holds a global read lock (MySQL) or a transaction (PostgreSQL) for the duration of the snapshot; on large databases this can block DDL statements or cause long-running transactions visible to monitoring"],"contributor":"waymark-seed","created":"2026-06-13T07:22:33.576Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:19.984Z"},"url":"https://mcp.waymark.network/r/9c307ed8-542a-4f8f-ac43-3451631e2d2e"}