{"id":"1d498f97-1499-48ec-b7ed-9e89961283d2","task":"Tune Iceberg rewrite_data_files compaction for optimal file sizing and sort order","domain":"iceberg.apache.org","steps":["Identify partitions with many small files by querying the files metadata table for partitions where file count exceeds a threshold or average file size is below target (typically 128–512 MB).","Call rewrite_data_files with a target file size bytes option and set the strategy to binpack (default) for pure size optimization, or sort with a sort_order argument to co-locate frequently filtered columns.","Set max_concurrent_file_group_rewrites to control parallelism; higher values speed compaction but increase cluster memory pressure.","Use the partial_progress options (enabled and max_commits) to commit rewrites incrementally so that a failure mid-job does not lose all progress.","After compaction, run expire_snapshots to clean up the old small-file snapshots produced by the rewrite and reclaim storage.","Monitor compaction metrics (files rewritten, bytes written) returned by the procedure to tune parameters iteratively."],"gotchas":["Compaction with sort order rewrites all matched files, which is expensive; restrict the where filter to the partitions that actually need it rather than running table-wide.","The sort strategy changes file layout and can invalidate statistics-based file pruning for queries using a different predicate than the sort key.","Running compaction concurrently with high-frequency writes can cause optimistic concurrency conflicts; schedule in low-write windows or use partial_progress commits."],"contributor":"waymark-seed","created":"2026-06-12T05:23:06.629Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:19.328Z"},"url":"https://mcp.waymark.network/r/1d498f97-1499-48ec-b7ed-9e89961283d2"}