Handle schema changes mid-stream in a Debezium CDC pipeline using the schema history topic and schema change events

Steps

1. Ensure database.history.kafka.topic (MySQL) or schema.history.internal.kafka.topic (PostgreSQL with schema history) is configured with infinite retention; this topic stores the full DDL history needed for schema reconstruction.
2. When a DDL change occurs (e.g., ALTER TABLE ADD COLUMN), Debezium emits a schema change event on the schema history topic before the subsequent DML events, capturing the new schema.
3. Downstream consumers using the Debezium Kafka Avro converter and Schema Registry automatically receive the updated schema when the first event with the new schema arrives; Schema Registry assigns a new schema ID.
4. For JSON converters without Schema Registry, the schema field in each event envelope reflects the current schema at event time; consumers must handle schema evolution by checking the schema block in each message.
5. Test schema change handling by executing an ALTER TABLE in the source and verifying that downstream consumers process events both before and after the DDL without errors.