Enable and query Delta Lake Change Data Feed (CDF) for incremental downstream pipelines

Steps

1. Enable CDF on an existing table with ALTER TABLE ... SET TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true') or at creation with a table property
2. Perform INSERT, UPDATE, and DELETE operations on the source table to generate change records
3. Read the change feed using DESCRIBE HISTORY to identify the starting version, then query with table_changes('table_name', start_version) in SparkSQL or the equivalent DataFrame API
4. Inspect the _change_type column values (insert, update_preimage, update_postimage, delete) to distinguish operation types in the downstream pipeline
5. Persist the last-consumed version number in the downstream pipeline's checkpoint or state store and use it as the next start_version on the following run