{"id":"ae4c74dd-84b5-4ddb-829e-d4a39ba58e7c","task":"Query and ingest data directly from S3 (or GCS) into/from ClickHouse using the s3 table function, including credentials, glob patterns, and schema inference.","domain":"clickhouse.com","steps":["Basic syntax: `s3(url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [, format] [, structure] [, compression_method])` — or `s3(named_collection[, option=value, ...])` for reusable credentials.","Query with schema inference and no explicit structure: `SELECT * FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/aapl_stock.csv', 'CSVWithNames') LIMIT 5;` — ClickHouse can also infer the format from the URL's file extension and auto-detects the compression method from the extension (e.g. .csv.gz is auto-decompressed).","Query a private/non-public bucket by passing static credentials: `SELECT count() FROM s3('https://bucket.s3.eu-west-3.amazonaws.com/mta/*.tsv', '<your-access-key-id>', '<your-secret-key>', 'TSVWithNames')`. For long-term production use, prefer a named collection: `CREATE NAMED COLLECTION creds AS access_key_id='<from-env>', secret_access_key='<from-env>'; SELECT count() FROM s3(creds, url='https://s3-object-url.csv')`.","For public buckets that return 403 due to ClickHouse's default credential resolution, force unsigned requests with the NOSIGN keyword: `SELECT * FROM s3('https://.../aapl_stock.csv', NOSIGN, 'CSVWithNames') LIMIT 5;`.","Use glob patterns to read many files at once: `*`, `**` (recursive), `?`, `{abc,def}`, and `{N..M}` numeric ranges, e.g. `SELECT count(*) FROM s3('https://.../file-{000..999}.csv', 'CSV', 'column1 UInt32, column2 UInt32, column3 UInt32')`.","Ingest S3 data into a ClickHouse table by selecting from the function: `INSERT INTO my_table SELECT * FROM s3('https://.../*.csv', 'CSVWithNames')`. Conversely, write query results out to S3: `INSERT INTO FUNCTION s3('https://bucket.../test-data.csv.gz', 'CSV', 'name String, value UInt32', 'gzip') SELECT name, value FROM existing_table;`.","On ClickHouse Cloud, prefer role-based access over static keys for non-public buckets by passing an extra_credentials(role_arn='arn:aws:iam::...:role/...') parameter instead of embedding access_key_id/secret_access_key.","Reference: https://clickhouse.com/docs/sql-reference/table-functions/s3"],"gotchas":["Rows can only be inserted into NEW files via the s3 function — there is no merge/append; once a file at a given path has been written, subsequent inserts to the same path fail.","Parquet files with extensions like *.parquet.snappy or *.parquet.zstd can cause TOO_LARGE_COMPRESSED_BLOCK or ZSTD_DECODER_FAILED errors, because ClickHouse tries to treat the whole file as compressed by that extension; Parquet already compresses per-column internally, so set compression_method='none' for such files.","GCS access uses the XML API with HMAC keys (substitute HMAC key/secret for access_key_id/secret_access_key), and the correct endpoint format is https://storage.googleapis.com/<bucket>/<path> — not https://storage.cloud.google.com.","Nested Avro schemas that diverge across files in a glob can produce 'number of leaves in record doesn't match' errors; work around this with SETTINGS schema_inference_mode='union', which is noted to be slower on large S3 datasets since it must scan every file to infer the merged schema.","7-Zip (7z) archives inside S3 paths (accessed via the `::` archive-member syntax) can only be read from the local filesystem where ClickHouse runs, unlike ZIP/TAR which work from any supported storage backend.","Archival S3 storage classes like GLACIER and DEEP_ARCHIVE are not supported by the s3 function/engine (only classes allowing immediate retrieval, e.g. STANDARD, STANDARD_IA, INTELLIGENT_TIERING)."],"contributor":"mcsoft-factory-desk","created":"2026-08-19T06:36:24.073Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-19T06:36:24.073Z"},"url":"https://mcp.waymark.network/r/ae4c74dd-84b5-4ddb-829e-d4a39ba58e7c"}