{"id":"abf5faa0-9fa7-47ba-a4d0-15545454829e","task":"Achieve exactly-once writes with the BigQuery Storage Write API using a committed-type stream","domain":"data-engineering","steps":["Don't rely on the default stream for exactly-once — it's documented as at-least-once; use it only when idempotency doesn't matter or is handled downstream.","For exactly-once, call CreateWriteStream with type COMMITTED to create your own stream instead of writing to the default stream.","On each AppendRows call, supply the expected next offset for that stream — the server only accepts the write if the offset matches, which is what gives exactly-once semantics within the stream.","Use GetWriteStream to check a stream's current state/offset if a client needs to resume after a crash rather than guessing where it left off.","Reserve PENDING streams for batch-style loads where rows must stay invisible until you call FinalizeWriteStream and explicitly commit — pending streams must be committed within 24 hours.","Only use the BUFFERED stream type (row-level commit via flush) if integrating with the Apache Beam BigQuery I/O connector — it's called out as an advanced type, not for general use."],"gotchas":["\"Default stream\" and \"committed stream\" sound similar but differ in guarantees — the default stream is at-least-once even though it uses committed-type semantics; exactly-once requires a self-managed stream with offset tracking.","PENDING streams not finalized and committed within 24 hours are effectively lost — don't treat PENDING as a durable staging area with no deadline."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/abf5faa0-9fa7-47ba-a4d0-15545454829e"}