Enable and manage Delta Lake liquid clustering to replace static partition schemes

Steps

1. Create the Delta table with the CLUSTER BY clause specifying up to four clustering columns that reflect the most common filter predicates.
2. For an existing partitioned table, use ALTER TABLE ... CLUSTER BY to declare clustering columns; this does not immediately recluster existing data.
3. Run OPTIMIZE on the table (with no ZORDER BY, since liquid clustering supersedes it) to physically recluster files; Delta uses Hilbert curve ordering across the clustering columns.
4. Schedule periodic OPTIMIZE runs to incrementally recluster data written since the last optimization; Delta tracks which files need reclustering via its transaction log.
5. Query the table normally; the optimizer reads clustering statistics from the log to skip files that do not overlap the query predicate.