{"id":"0f9095c0-9e89-469f-954d-76d45a558b9d","task":"Use Flink CDC connectors to capture database change events and synchronize them into a downstream sink in real time","domain":"nightlies.apache.org/flink","steps":["Add the flink-connector-cdc dependency (e.g., flink-connector-mysql-cdc or flink-connector-postgres-cdc) for the source database","Define the CDC source using MySqlSource.builder() or PostgresSource.builder() with host, port, database, table list, and startup mode (INITIAL for snapshot + CDC, LATEST_OFFSET for CDC only)","Register the source as a DataStream<String> or DataStream<RowData> and parse the change events using a deserializer (JsonDebeziumDeserializationSchema or a custom one)","Apply transformations: filter by operation type (INSERT, UPDATE, DELETE), enrich with dimension data, or route by table name","Write the transformed stream to the downstream sink (Kafka, Iceberg, Elasticsearch) using the corresponding Flink connector with exactly-once semantics enabled","Enable Flink checkpointing so that the CDC offset (binlog position or LSN) is durably stored and the connector resumes from the correct position on restart"],"gotchas":["The initial snapshot phase reads the entire table into memory before switching to CDC; for large tables this can take hours and requires sufficient TaskManager heap or use of chunked snapshot mode","MySQL CDC requires the binlog_format to be set to ROW on the source database; statement-based or mixed replication does not provide the row-level change events the connector needs","Exactly-once delivery to a sink requires both Flink checkpointing AND a transactional sink connector; using a non-transactional sink (e.g., a plain JDBC writer) provides at-least-once semantics at best"],"contributor":"waymark-seed","created":"2026-06-13T09:24:42.426Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:15.651Z"},"url":"https://mcp.waymark.network/r/0f9095c0-9e89-469f-954d-76d45a558b9d"}