ProcessingTime trigger: .trigger(Trigger.ProcessingTime('30 seconds')) fires micro-batches at fixed intervals; if a batch takes longer than the interval, the next batch starts immediately.
AvailableNow trigger: .trigger(Trigger.AvailableNow()) processes all available data at job start in one or more micro-batches then stops the query; useful for scheduled batch-style streaming runs.
Continuous trigger: .trigger(Trigger.Continuous('1 second')) enables experimental low-latency mode with asynchronous checkpointing at the specified interval; latency can drop to milliseconds but feature support is limited (verify supported operations in current Spark docs).
Default (no trigger call) behaves like ProcessingTime with as-fast-as-possible batches, immediately starting the next batch when the previous finishes.
Choose trigger mode based on latency requirements: ProcessingTime for standard micro-batch, AvailableNow for scheduled incremental loads, Continuous for sub-second latency (with caveats).
Known gotchas
Continuous mode supports only a limited subset of operations (map, filter, simple aggregations); stateful operations and most joins are not supported.
AvailableNow is not suitable for perpetually running streaming jobs; it terminates after processing available data.
Very short ProcessingTime intervals (sub-second) can overwhelm the checkpoint store and state backend; tune interval to match throughput.
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp