{"id":"c81e0bff-739c-413b-a8b9-b76b81931c64","task":"Read and write hive-partitioned Parquet datasets in a DuckDB pipeline","domain":"data-engineering","steps":["Read partitioned data with SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true) — DuckDB auto-detects hive-style 'key=value' folder naming and enables partitioning automatically, but set the flag explicitly in scripts.","Set hive_partitioning = false explicitly if partition columns should be excluded from the result set rather than surfaced as regular columns.","Rely on automatic filter pushdown on partition keys — a WHERE clause on a partition column prunes files at scan time with no extra config.","Write partitioned output with COPY <table> TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)).","Since PARTITION_BY can't take expressions directly, pre-compute any derived partition columns in a subquery first, e.g. COPY (SELECT *, year(timestamp) AS year FROM services) TO 'test' (PARTITION_BY (year, month))."],"gotchas":["PARTITION_BY rejecting expressions is easy to trip over — writing PARTITION_BY (year(timestamp)) directly fails; the derived column must be materialized in the SELECT first.","Auto-detection of hive partitioning means a directory with incidentally 'key=value'-shaped folder names gets treated as partitioned — verify a path glob isn't picking up unrelated folders matching the pattern."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/c81e0bff-739c-413b-a8b9-b76b81931c64"}