{"id":"7bb358be-d466-4e1b-9113-ee7ae2d3881f","task":"Use DuckDB with the httpfs extension as a lightweight transformation engine over Parquet files in S3","domain":"duckdb.org","steps":["Run INSTALL httpfs; LOAD httpfs; once per DuckDB environment to enable S3-compatible object storage access.","Create a secret for S3 credentials with CREATE SECRET (TYPE s3, KEY_ID '<key>', SECRET '<secret>', REGION '<region>'), or use PROVIDER credential_chain to pick up credentials automatically from the environment or an IAM role.","Query Parquet files directly from S3 with SELECT * FROM read_parquet('s3://bucket/path/*.parquet'), including reading a glob of multiple files as a single relation.","Run transformation SQL (joins, aggregations, filters) directly against the S3-backed Parquet relations and write results back out with COPY ... TO 's3://bucket/output.parquet' (FORMAT PARQUET).","Embed this pattern as a lightweight pipeline step (e.g. a scheduled script or orchestrator task) for ad hoc or small-to-medium transformation jobs that don't warrant a full Spark or warehouse job."],"gotchas":["DuckDB executes locally (single-node); very large S3 datasets that exceed available memory/disk on the executing machine won't scale the way a distributed engine would, even though pushdown-friendly Parquet reads can still be efficient.","CREATE SECRET credentials are process/session-scoped by default (or persisted to disk depending on secret persistence settings); treat persisted secrets on shared machines as a credential-exposure risk.","httpfs must be explicitly installed and loaded in every fresh DuckDB session/process; forgetting LOAD httpfs after a prior INSTALL results in read_parquet('s3://...') failing."],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/7bb358be-d466-4e1b-9113-ee7ae2d3881f"}