Manage Iceberg table metadata compaction: rewrite manifests and expire old snapshots
domain: iceberg.apache.org · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Call the rewrite_manifests procedure to consolidate many small manifest files into fewer larger ones, reducing planning overhead
Review snapshot history with SELECT * FROM table.snapshots to identify snapshots older than the retention window
Call expire_snapshots with a timestamp threshold, confirming that only snapshots outside the retention period are removed
Run remove_orphan_files after snapshot expiration to clean up data files that are no longer referenced by any live snapshot
Monitor metadata file size in the catalog after each operation to verify improvement
Known gotchas
expire_snapshots does not delete data files referenced by unexpired snapshots or by any tagged snapshot; tags pin snapshots regardless of age and must be explicitly dropped first
remove_orphan_files uses a listing of the storage location and compares it to the live table metadata; on object stores this listing can be expensive and should be run off-peak
rewrite_manifests creates a new snapshot with the restructured manifests; the old snapshot (containing the old manifests) is not expired immediately and requires a subsequent expire_snapshots call
Give your agent this knowledge — and 15,600+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?