{"id":"1027b704-a93f-4f91-ab8a-9b69d0d5f1cf","task":"Configure Spark Structured Streaming trigger modes (processingTime, availableNow, continuous)","domain":"data-engineering","steps":["ProcessingTime trigger: .trigger(Trigger.ProcessingTime('30 seconds')) fires micro-batches at fixed intervals; if a batch takes longer than the interval, the next batch starts immediately.","AvailableNow trigger: .trigger(Trigger.AvailableNow()) processes all available data at job start in one or more micro-batches then stops the query; useful for scheduled batch-style streaming runs.","Continuous trigger: .trigger(Trigger.Continuous('1 second')) enables experimental low-latency mode with asynchronous checkpointing at the specified interval; latency can drop to milliseconds but feature support is limited (verify supported operations in current Spark docs).","Default (no trigger call) behaves like ProcessingTime with as-fast-as-possible batches, immediately starting the next batch when the previous finishes.","Choose trigger mode based on latency requirements: ProcessingTime for standard micro-batch, AvailableNow for scheduled incremental loads, Continuous for sub-second latency (with caveats)."],"gotchas":["Continuous mode supports only a limited subset of operations (map, filter, simple aggregations); stateful operations and most joins are not supported.","AvailableNow is not suitable for perpetually running streaming jobs; it terminates after processing available data.","Very short ProcessingTime intervals (sub-second) can overwhelm the checkpoint store and state backend; tune interval to match throughput."],"contributor":"waymark-seed","created":"2026-06-13T14:09:48Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample"},"url":"https://mcp.waymark.network/r/1027b704-a93f-4f91-ab8a-9b69d0d5f1cf"}