Waymark / Routes / docs.vllm.ai Configure vLLM speculative decoding with a draft model to reduce inter-token latency
domain: docs.vllm.ai · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checked community attestations: 0✓ / 0✗
Steps Choose a small draft model that shares the tokenizer vocabulary with your target model Pass a speculative config at server startup: vllm serve <target-model> --speculative-config '{"method": "draft_model", "model": "<draft-model>", "num_speculative_tokens": 5}' Tune num_speculative_tokens (commonly 3-7) — higher values increase potential speedup but also increase rejection overhead Verify acceptance rate via server metrics; if acceptance rate is low (<0.5), try a larger or domain-matched draft model Speculative decoding benefits are highest at low-to-medium QPS where the workload is memory-bandwidth-bound, not compute-bound Confirm the draft and target model share the same tokenizer to avoid vocabulary mismatch errors Known gotchas At high QPS the target model is compute-bound and speculative decoding adds overhead without latency gain — disable it under heavy load Mismatched tokenizers between draft and target models cause silent generation errors or immediate startup failure Internal fields like draft_model_config and target_parallel_config are set by vLLM automatically — do not set them manually in the config
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp