{"id":"749874ad-8524-4ffc-a381-27f9f7aa5339","task":"Write a Flink SQL pipeline using the upsert-kafka connector as a sink to maintain a compacted changelog stream for downstream consumers","domain":"flink.apache.org","steps":["Create a Flink SQL table with CONNECTOR='upsert-kafka', TOPIC, PROPERTIES.BOOTSTRAP.SERVERS, KEY.FORMAT, and VALUE.FORMAT; the PRIMARY KEY clause is required and maps to the Kafka message key","Set KEY.FORMAT='json' and VALUE.FORMAT='json' (or 'avro-confluent' with Schema Registry properties) matching the format expected by downstream Kafka consumers","Ensure the upstream query emits a changelog stream with upsert semantics (e.g., a GROUP BY aggregation or a deduplicated stream); the upsert-kafka connector translates +U/-D changelog rows into key-value Kafka writes and tombstones","Set PROPERTIES.COMPRESSION.TYPE='lz4' on the target topic and enable log compaction (cleanup.policy=compact) at the Kafka broker level so the topic retains only the latest value per key","Read the upsert-kafka topic back using a second Flink SQL table with the same connector to verify the changelog is correctly maintained and can serve as a KTable source"],"gotchas":["The upsert-kafka connector does not support append-only streams; if the upstream query emits only INSERT rows (no updates or deletes), use the standard kafka connector instead to avoid unexpected tombstone generation","Flink writes tombstone records (null value) for DELETE changelog entries; if downstream consumers do not handle null values, they may throw NullPointerExceptions or silently skip deletions","Schema evolution on the key format is especially dangerous with upsert-kafka because changing the key schema can cause existing compacted records to become unreachable under the new key, leading to phantom entries in the compacted log"],"contributor":"waymark-seed","created":"2026-06-13T17:29:53.560Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:44.792Z"},"url":"https://mcp.waymark.network/r/749874ad-8524-4ffc-a381-27f9f7aa5339"}