{"id":"70cab299-ba0e-4007-9e6c-daa2e142ca5f","task":"Generate and use Iceberg Puffin NDV statistics files to improve query planner decisions","domain":"iceberg.apache.org","steps":["Run the engine's statistics-computation procedure (e.g. Spark's ANALYZE TABLE COMPUTE STATISTICS) to produce a Theta-sketch (apache-datasketches-theta-v1) NDV blob per column.","Confirm the resulting Puffin file is registered in the table metadata's statistics field, tied to the snapshot ID it was computed against.","Query the table's metadata to list registered statistics files, the blob types they contain, and which columns/snapshot they apply to.","Re-run statistics computation after significant data changes, since a stats file bound to an old snapshot is not automatically reused for later snapshots.","Compare query plans before and after stats registration to confirm the engine's cost-based optimizer is actually consuming the NDV values."],"gotchas":["Statistics files are snapshot-scoped; stale entries pointing at superseded snapshots should be cleaned up rather than assumed to still apply.","Not all query engines read or write Puffin statistics yet, so verify engine-specific support before relying on this for cost-based optimization.","Only blobs of the reserved standard type are understood by generic readers; custom blob types will be silently ignored by engines expecting the standard NDV sketch."],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/70cab299-ba0e-4007-9e6c-daa2e142ca5f"}