{"id":"70085b8d-3848-4071-a5fc-d67a99ffff9b","task":"Quantize an LLM to 4-bit for inference using bitsandbytes with Hugging Face Transformers (BitsAndBytesConfig)","domain":"ml-ops","steps":["Install the stack: pip install --upgrade transformers accelerate bitsandbytes","Build a 4-bit config: from transformers import BitsAndBytesConfig; quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)","Load the model with it: model = AutoModelForCausalLM.from_pretrained(model_id, device_map=\"auto\", quantization_config=quantization_config)","Check the memory savings with model.get_memory_footprint()","Run inference as usual with generate() — weights are stored in 4-bit but computation runs in the dtype set by bnb_4bit_compute_dtype","Optionally push the quantized model with model.push_to_hub(...), or later call model.dequantize() if you need full precision back"],"gotchas":["NF4/FP4 4-bit quantization requires NVIDIA Pascal (GTX 10-series/P100) or newer GPUs per HF's hardware compatibility table (bitsandbytes also has CUDA, Intel XPU, Intel Gaudi/HPU, and CPU backends)","8/4-bit quantized weights only support training of 'extra' parameters (e.g. LoRA adapters via PEFT/QLoRA) — you cannot directly fine-tune the quantized base weights themselves","dequantize() can restore original precision but may cause quality loss and requires enough GPU memory to hold the full-precision model"],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/70085b8d-3848-4071-a5fc-d67a99ffff9b"}