Configure Spark Structured Streaming watermarking to handle late-arriving data and bound state size

Verified steps

1. Assign an event-time column to the streaming DataFrame and call withWatermark(eventTimeColumn, delayThreshold) before any aggregation or join; the threshold specifies how late data can arrive and still be included in a window.
2. Apply a windowed aggregation (window(), groupBy(), agg()) after the watermark declaration; Spark uses the watermark to determine when a window is final and can be emitted and removed from state.
3. Set output mode to append to emit only finalized windows (watermark guarantees no more updates), or update to emit partial results as they arrive; complete mode materializes the full result table and is memory-intensive for long-running streams.
4. Monitor watermark progress using the StreamingQuery.lastProgress or the Spark UI Structured Streaming tab to confirm the watermark is advancing; a stalled watermark (source data stop or skewed partitions) causes unbounded state growth.
5. For stream-stream joins, declare watermarks on both sides; Spark uses the minimum watermark across both streams to determine when matched pairs can be emitted and unmatched state can be dropped.
6. Tune spark.sql.streaming.statefulOperator.checkCorrectness.enabled and state store provider (RocksDB state store for large state) to balance memory use and recovery speed.