Create a Google Cloud Pub/Sub BigQuery subscription to write messages directly into a BigQuery table
domain: cloud.google.com · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Create or identify an existing BigQuery table whose schema is compatible with your Pub/Sub message structure (or plan to use the topic schema if you want Pub/Sub to write structured columns automatically)
Grant the Pub/Sub service account the roles/bigquery.dataEditor role (and roles/bigquery.metadataViewer if needed) on the target BigQuery dataset so it can write rows
Create a subscription of type BigQuery subscription, specifying the target project, dataset, and table, via the Cloud Console, gcloud pubsub subscriptions create --bigquery-table, or the API
Understand that Pub/Sub uses the BigQuery Storage Write API under the hood to batch-write incoming messages as rows into the table
Configure a dead-letter topic on the BigQuery subscription to capture messages that fail to write to BigQuery; inspect the CloudPubSubDeadLetterSourceDeliveryErrorMessage attribute on forwarded messages to see the failure reason
Monitor subscription metrics for negative acknowledgments, which occur whenever a batch write to BigQuery fails, and iterate on schema compatibility until writes succeed consistently
Known gotchas
The BigQuery table's schema must meet Pub/Sub's compatibility requirements (column types/names aligned with the message JSON or the topic's defined schema) — mismatches cause per-message write failures rather than a subscription-level error
This is a fully-managed alternative to running your own Dataflow pipeline for Pub/Sub-to-BigQuery streaming — it trades pipeline flexibility (custom transforms, windowing) for operational simplicity, so it's not a drop-in replacement for all Dataflow use cases
Failed writes are negatively acknowledged (redelivered) rather than dropped by default, so a persistently malformed message can loop until you configure a dead-letter topic to break the cycle
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?