Consume Delta Lake Change Data Feed to build downstream incremental pipelines

Steps

1. Enable Change Data Feed on the source table by setting the delta.enableChangeDataFeed table property to true (ALTER TABLE or in CREATE TABLE TBLPROPERTIES).
2. Read changes using table_changes in SQL or the readChangeFeed option in Spark, providing a starting version or timestamp and optionally an ending version.
3. Filter the _change_type column (insert, update_preimage, update_postimage, delete) to apply the appropriate upsert or delete logic to the target table.
4. In Structured Streaming mode, set readChangeFeed to true and startingVersion to latest or a checkpoint version; the stream will emit new change rows as data is written.
5. Propagate the _commit_version and _commit_timestamp metadata columns to the sink if the downstream system needs ordering or deduplication guarantees.
6. Checkpoint the last processed version in the downstream system and use it as startingVersion on restart to avoid reprocessing.