configure triton inference server sequence batching for a stateful model
domain: docs.nvidia.com/deeplearning/triton-inference-server · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Add a sequence_batching section to the model's config.pbtxt to opt the model into the sequence batcher instead of the default dynamic batcher
Configure control inputs so Triton can signal sequence start, end, ready, and correlation ID to the model, since a stateful model must accept these control tensors
Ensure the client sends the same correlation ID on every request in a sequence and correctly marks start/end flags so Triton routes the whole sequence to one model instance
Set an appropriate sequence timeout so abandoned sequences release their reserved batch slot
Only rely on the iterative sequences sub-feature (single request processed over multiple scheduling iterations) if acceptable, since it is documented as provisional
Known gotchas
The sequence batcher requires all requests in a sequence to land on the same model instance to preserve state — misconfigured correlation IDs corrupt state across sequences
Iterative sequences are explicitly documented as provisional/likely to change — avoid long-term production dependencies on it without pinning the Triton version
Sequence batching is configured per-model and requires the model to implement the expected control-input tensors, or it fails to load
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?