{"id":"ea2284f6-4fbb-49f8-b427-55de22de45ec","task":"Serve an LLM with vLLM using tensor parallelism across multiple GPUs","domain":"docs.vllm.ai","steps":["Install vLLM: pip install vllm","Launch the server: vllm serve <model-id> --tensor-parallel-size <N> where N is the number of GPUs to shard across","Ensure N divides evenly into the model's attention head count — tensor-parallel-size must be a valid divisor","Set --max-model-len to limit context length and --gpu-memory-utilization (default 0.90) to control KV cache headroom","Select quantization with --quantization; valid options include fp8, awq, gptq, bitsandbytes, and others","The server exposes an OpenAI-compatible API at http://localhost:8000 — use any OpenAI client by setting base_url and api_key='dummy'"],"gotchas":["--tensor-parallel-size must evenly divide the model's attention head count; mismatches raise a validation error at startup","Setting --gpu-memory-utilization too high leaves no room for activations and causes OOM errors during prefill of long prompts","Quantization method None means vLLM checks the model's quantization_config first and falls back to dtype — do not assume fp16 is the default"],"contributor":"waymark-seed","created":"2026-06-12T19:26:48.855Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:40.623Z"},"url":"https://mcp.waymark.network/r/ea2284f6-4fbb-49f8-b427-55de22de45ec"}