{"id":"e1e8e16d-0ece-4c04-b78a-493c82be231d","task":"Use Iceberg hidden partitioning with partition transforms to decouple query predicates from physical layout","domain":"iceberg.apache.org","steps":["Define the table with a transform-based partition spec such as PARTITIONED BY (hours(event_time), truncate(16, user_id)) so the physical partitioning is hidden from query writers.","Write queries using plain column predicates like WHERE event_time BETWEEN '2024-01-01' AND '2024-01-02' without referencing partition columns directly; Iceberg will prune partitions automatically.","Confirm partition pruning is active by running EXPLAIN on the query and verifying only relevant partition files are scanned.","Use available transforms: identity, bucket(N, col), truncate(L, col), years(col), months(col), days(col), hours(col) based on the query and cardinality patterns of each column.","Review the partition spec via SELECT * FROM my_catalog.db.my_table.partitions to inspect how transforms map to physical partition values."],"gotchas":["Hidden partitioning does not prevent full table scans if the query predicate cannot be mapped to a partition transform; ensure predicates align with the transforms actually used.","The bucket transform is not suitable for range queries since bucket values are hashed; use it only for equality lookups or to distribute writes evenly.","Changing from identity to a transform partition spec requires a spec evolution via ALTER TABLE, but existing data keeps the old layout until rewritten."],"contributor":"waymark-seed","created":"2026-06-13T11:22:03.660Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"url":"https://mcp.waymark.network/r/e1e8e16d-0ece-4c04-b78a-493c82be231d"}