{"id":"4fb5d269-5f6a-4295-ba87-3fdcce2df3d0","task":"quantize a hugging face model to 4-bit with autoawq and save it for deployment","domain":"huggingface.co/docs/transformers/quantization/awq","steps":["Install autoawq and load the base model with AutoAWQForCausalLM.from_pretrained()","Define a quant_config (bits, group size, zero-point) and run model.quantize(tokenizer, quant_config=quant_config), which requires a calibration data pass","Save the quantized weights with model.save_quantized(quant_path), optionally passing safetensors=True and a shard_size","Save the tokenizer alongside the quantized model with tokenizer.save_pretrained(quant_path) so the artifact is self-contained","Validate the quantized model's outputs against the fp16 baseline on a held-out prompt set before deploying","Push the quantized checkpoint to a model registry or the Hugging Face Hub, tagging it clearly as AWQ-quantized for downstream serving tools to auto-detect"],"gotchas":["Quantization requires a representative calibration dataset — an unrepresentative or too-small calibration set degrades accuracy silently","AutoAWQ's save_quantized wraps transformers' save_pretrained, so the config.json quantization metadata must be present for serving frameworks to auto-detect the format","Group size and bit-width choices trade off compression vs accuracy — verify parity on your own eval set rather than assuming published benchmarks transfer"],"contributor":"waymark-seed","created":"2026-07-10T03:38:47.861Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/4fb5d269-5f6a-4295-ba87-3fdcce2df3d0"}