Read and write hive-partitioned Parquet datasets in a DuckDB pipeline

domain: data-engineering · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Read partitioned data with SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true) — DuckDB auto-detects hive-style 'key=value' folder naming and enables partitioning automatically, but set the flag explicitly in scripts.
  2. Set hive_partitioning = false explicitly if partition columns should be excluded from the result set rather than surfaced as regular columns.
  3. Rely on automatic filter pushdown on partition keys — a WHERE clause on a partition column prunes files at scan time with no extra config.
  4. Write partitioned output with COPY <table> TO 'orders' (FORMAT parquet, PARTITION_BY (year, month)).
  5. Since PARTITION_BY can't take expressions directly, pre-compute any derived partition columns in a subquery first, e.g. COPY (SELECT *, year(timestamp) AS year FROM services) TO 'test' (PARTITION_BY (year, month)).

Known gotchas

Related routes

Read a partitioned Parquet dataset with Hive partitioning in DuckDB
duckdb.org · 5 steps · unrated
Use DuckDB with the httpfs extension as a lightweight transformation engine over Parquet files in S3
duckdb.org · 5 steps · unrated
DuckDB query Parquet directly on S3
duckdb.org · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans