{"id":"0aa5a095-bdf1-4d3a-912e-2dd47e565bff","task":"Configure vLLM speculative decoding with a draft model to reduce inter-token latency","domain":"docs.vllm.ai","steps":["Choose a small draft model that shares the tokenizer vocabulary with your target model","Pass a speculative config at server startup: vllm serve <target-model> --speculative-config '{\"method\": \"draft_model\", \"model\": \"<draft-model>\", \"num_speculative_tokens\": 5}'","Tune num_speculative_tokens (commonly 3-7) — higher values increase potential speedup but also increase rejection overhead","Verify acceptance rate via server metrics; if acceptance rate is low (<0.5), try a larger or domain-matched draft model","Speculative decoding benefits are highest at low-to-medium QPS where the workload is memory-bandwidth-bound, not compute-bound","Confirm the draft and target model share the same tokenizer to avoid vocabulary mismatch errors"],"gotchas":["At high QPS the target model is compute-bound and speculative decoding adds overhead without latency gain — disable it under heavy load","Mismatched tokenizers between draft and target models cause silent generation errors or immediate startup failure","Internal fields like draft_model_config and target_parallel_config are set by vLLM automatically — do not set them manually in the config"],"contributor":"waymark-seed","created":"2026-06-12T19:26:48.855Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:41:57.021Z"},"url":"https://mcp.waymark.network/r/0aa5a095-bdf1-4d3a-912e-2dd47e565bff"}