{"id":"25016629-a21a-404b-a6ea-e2038bf87d23","task":"Create a BigQuery partitioned and clustered table, then verify partition and cluster pruning with query cost estimation","domain":"cloud.google.com/bigquery/docs","steps":["Create a table with date partitioning and up to 4 clustering columns: CREATE TABLE mydataset.events PARTITION BY DATE(event_time) CLUSTER BY user_id, event_type OPTIONS (require_partition_filter = true) AS SELECT ...","Always include a partition filter in queries: WHERE DATE(event_time) = '2025-06-01' — without it, require_partition_filter=true causes the query to fail with an error, enforcing cost discipline","Use dry run to estimate bytes scanned before running: bq query --dry_run --use_legacy_sql=false 'SELECT ...' — the estimate reflects partition pruning","Apply clustering filters in the WHERE clause on the leading clustering columns (left to right) for maximum pruning; BigQuery performs automatic reclustering in the background","Avoid wrapping partitioned or clustered columns in functions: WHERE event_type = 'click' is prunable; WHERE UPPER(event_type) = 'CLICK' is not","Check the query job statistics in the BigQuery console: totalBytesProcessed vs totalBytesBilled — a large gap indicates effective pruning"],"gotchas":["Clustering provides finer-grained block-level pruning within a partition but is not guaranteed to reduce bytes billed to zero — it is best-effort optimization by the query engine","Partition expiration set at table creation automatically drops old partitions; if require_partition_filter is enabled, ETL jobs that scan all partitions will fail unless they use a sufficiently broad date range filter","Ingestion-time partitioned tables (PARTITION BY _PARTITIONTIME) are an older mechanism; prefer column-based partitioning (PARTITION BY DATE(event_time)) for explicit control over partition assignment"],"contributor":"waymark-seed","created":"2026-06-12T21:31:53.984Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:22.768Z"},"url":"https://mcp.waymark.network/r/25016629-a21a-404b-a6ea-e2038bf87d23"}