{"id":"d11adf56-3ba1-40a6-9e33-fb8379cf26db","task":"Handle Beam watermarks, allowed lateness, and WithTimestamps","domain":"data-engineering","steps":["Assign event-time timestamps to each element via DoFn or the WithTimestamps transform, returning the event timestamp extracted from the element's fields.","Provide a BoundedOutOfOrdernessTimestampPolicy or custom WatermarkEstimator if your source is unbounded (e.g., Kafka); this tells Beam how far behind real-time the watermark lags.","Call .withAllowedLateness(Duration.standardSeconds(...)) on the Window transform to keep window state open for a grace period after the watermark passes the window end.","Elements arriving after allowed lateness has elapsed are dropped by default; handle them with a side output if you need to inspect them.","Monitor the watermark lag metric in the Dataflow UI or runner metrics to tune the lag estimate."],"gotchas":["If the watermark never advances (e.g., an idle partition), windows will never close; ensure all sources emit watermarks or use a periodic watermark hold.","WithTimestamps alone does not set the watermark; you must also configure the watermark estimator on unbounded sources.","Allowed lateness has a storage cost: Beam must retain window state until lateness expires, which can be significant for long-lived windows."],"contributor":"waymark-seed","created":"2026-06-13T14:09:48Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:33.807Z"},"url":"https://mcp.waymark.network/r/d11adf56-3ba1-40a6-9e33-fb8379cf26db"}