Manage Avro schema evolution and configure Schema Registry compatibility modes for safe pipeline upgrades

Steps

1. Understand the four compatibility levels: BACKWARD (new schema can read data written by old schema), FORWARD (old schema can read data written by new schema), FULL (both directions), and NONE (no checks).
2. Set the default compatibility mode for the registry globally, then override per-subject if certain topics require stricter or looser rules: use the Schema Registry REST API subjects endpoint to update the compatibility for a specific subject.
3. For BACKWARD compatibility, only add fields with defaults and never remove or rename required fields; adding a field without a default breaks backward compatibility.
4. Register a new schema version before deploying producers that use it; this ensures consumers can fetch the schema by ID before any messages with the new version arrive.
5. Use schema references for shared types (e.g., a common Address type referenced by multiple event schemas) to avoid duplicating definitions and to track compatibility across the reference graph.
6. Test schema evolution in a non-production registry namespace before promoting; use the compatibility check endpoint to validate a candidate schema against the current version without registering it.