docs.vllm.ai

serve a pre-quantized awq or gptq checkpoint with vllm

configure vllm continuous batching limits with max_num_seqs and max_num_batched_tokens

Serve an LLM through vLLM's OpenAI-compatible API server

Configure tensor and pipeline parallelism for multi-GPU vLLM serving

Configure vLLM speculative decoding with a draft model to reduce inter-token latency

Enforce structured JSON output from a vLLM server using guided decoding

Serve an LLM with vLLM using tensor parallelism across multiple GPUs

Enable automatic prefix caching in vLLM to reduce repeated-prompt latency

Deploy an LLM with vLLM using speculative decoding and automatic prefix caching for latency optimization

Serve LLMs with vLLM's OpenAI-compatible server