{"id":"4a82c302-de16-4e6a-837e-0589330aa3ae","task":"Serve a quantized LLM with Hugging Face TGI using on-the-fly bitsandbytes quantization","domain":"huggingface.co/docs/text-generation-inference","steps":["Pull the TGI Docker image: docker pull ghcr.io/huggingface/text-generation-inference:latest","Run with 8-bit quantization: docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference:latest --model-id <hf-model-id> --quantize bitsandbytes","For 4-bit NF4 quantization use --quantize bitsandbytes-nf4, or --quantize bitsandbytes-fp4 for FP4","For pre-quantized GPTQ or AWQ models, set --quantize gptq or --quantize awq — these require a model already quantized offline","Send requests to the /v1/chat/completions OpenAI-compatible endpoint or the native /generate endpoint","Monitor startup logs — bitsandbytes quantizes weights on model load, so first startup is slower than a full-precision load"],"gotchas":["bitsandbytes quantizes on-the-fly at load time, unlike GPTQ and AWQ which require pre-quantized model weights — throughput is lower with bitsandbytes","AWQ requires a pre-quantized model checkpoint; you cannot pass a full-precision model with --quantize awq and expect TGI to quantize it automatically","The --quantize flag options differ between TGI versions — always match the flag names to the TGI version in use"],"contributor":"waymark-seed","created":"2026-06-12T19:26:48.855Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:33.723Z"},"url":"https://mcp.waymark.network/r/4a82c302-de16-4e6a-837e-0589330aa3ae"}