{"id":"33b0aea0-d1a2-464f-b559-10bdf0acdeaa","task":"serve a pre-quantized awq or gptq checkpoint with vllm","domain":"docs.vllm.ai","steps":["Confirm the checkpoint on Hugging Face Hub is pre-quantized in AWQ or GPTQ/GPTQModel format by checking the model card","For AWQ, install the backend with pip install autoawq and load with LLM(model=..., quantization=\"auto_awq\") or pass --quantization awq to vllm serve","For GPTQ, install pip install auto-gptq (or use a GPTQModel-produced checkpoint) and pass --quantization gptq","Check the vLLM quantization docs page for the current list of supported methods and any Marlin/Machete kernel requirements for your GPU generation","Benchmark against the unquantized model to confirm the expected memory reduction and any throughput/accuracy tradeoff before promoting to production"],"gotchas":["vLLM does not bundle GPTQ/AWQ backends by default — the required pip package (auto-gptq or autoawq) must be installed separately or the server fails to load the model","GPTQModel-produced checkpoints get better performance on Ampere/Hopper GPUs via Marlin/Machete kernels than plain GPTQ; confirm which kernel path is actually selected","Passing the wrong --quantization value for the actual checkpoint format (e.g. gptq for an awq checkpoint) fails at load time — verify the model card before setting the flag"],"contributor":"waymark-seed","created":"2026-07-10T03:38:47.861Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/33b0aea0-d1a2-464f-b559-10bdf0acdeaa"}