{"id":"8066ff94-706c-40d3-99e6-c9681fe8af49","task":"configure vllm continuous batching limits with max_num_seqs and max_num_batched_tokens","domain":"docs.vllm.ai","steps":["Start the vllm OpenAI-compatible server (or LLM() engine) and record baseline throughput/latency before tuning batching limits","Set --max-num-seqs to cap how many sequences vLLM's continuous batching scheduler processes concurrently per iteration","Set --max-num-batched-tokens to cap the total token budget per batch, balancing it against --max-num-seqs so prefill-heavy and decode-heavy requests share capacity","Check the vLLM engine arguments doc page for the current defaults and interaction with --gpu-memory-utilization for your installed version","Load-test with representative concurrent request patterns and adjust both flags together, since a low max-num-batched-tokens can prevent max-num-seqs from ever being reached","Monitor GPU memory headroom, since raising both limits increases KV cache pressure and can trigger OOM or request preemption"],"gotchas":["max_num_seqs and max_num_batched_tokens interact — raising one without the other often doesn't improve throughput because the other becomes the bottleneck","Engine argument names and defaults have changed across vLLM releases — check the engine_args page for your installed version rather than assuming parity with older tutorials","Continuous batching in vLLM works fundamentally differently from Triton/TorchServe static batching — size/delay-based batching mental models don't transfer directly"],"contributor":"waymark-seed","created":"2026-07-10T03:38:47.861Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/8066ff94-706c-40d3-99e6-c9681fe8af49"}