Achieve exactly-once writes with the BigQuery Storage Write API using a committed-type stream

domain: data-engineering · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Don't rely on the default stream for exactly-once — it's documented as at-least-once; use it only when idempotency doesn't matter or is handled downstream.
  2. For exactly-once, call CreateWriteStream with type COMMITTED to create your own stream instead of writing to the default stream.
  3. On each AppendRows call, supply the expected next offset for that stream — the server only accepts the write if the offset matches, which is what gives exactly-once semantics within the stream.
  4. Use GetWriteStream to check a stream's current state/offset if a client needs to resume after a crash rather than guessing where it left off.
  5. Reserve PENDING streams for batch-style loads where rows must stay invisible until you call FinalizeWriteStream and explicitly commit — pending streams must be committed within 24 hours.
  6. Only use the BUFFERED stream type (row-level commit via flush) if integrating with the Apache Beam BigQuery I/O connector — it's called out as an advanced type, not for general use.

Known gotchas

Related routes

Compare BigQuery streaming inserts versus Storage Write API and implement the correct path for high-throughput ingestion
cloud.google.com · 6 steps · unrated
Implement Kafka exactly-once semantics using transactions
kafka.apache.org · 5 steps · unrated
Implement Flink exactly-once end-to-end semantics with a Kafka source and a transactional Kafka sink using two-phase commit
nightlies.apache.org/flink · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans