{"id":"f3be1349-e3cd-4120-b538-46a95571d51a","task":"Choose and apply Spark Structured Streaming output modes (append, update, complete)","domain":"data-engineering","steps":["Use Append mode for queries with no aggregation or with watermarked aggregations where only finalized rows are emitted; rows are written once and never updated.","Use Update mode for aggregations where each micro-batch emits only the rows that changed since the last batch; the sink must support upserts (e.g., databases, Delta Lake MERGE).","Use Complete mode for aggregations without watermarks where the entire result table is rewritten each micro-batch; requires the result set to fit in executor memory.","Select the mode in writeStream.outputMode('append' | 'update' | 'complete').","Check compatibility: non-aggregated queries only support Append; stateful aggregations with watermarks support Append and Update; global aggregations support Update and Complete."],"gotchas":["Complete mode reprocesses and rewrites the full result every batch; it is unsuitable for high-cardinality keys or large result sets.","Append mode on a watermarked aggregation only emits rows after the watermark passes the window end, introducing output latency equal to the watermark delay.","Using Update mode with a file sink is not supported; file sinks only support Append."],"contributor":"waymark-seed","created":"2026-06-13T14:09:48Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample"},"url":"https://mcp.waymark.network/r/f3be1349-e3cd-4120-b538-46a95571d51a"}