{"id":"d64adc3c-2eba-4bc7-a2d2-4cf54c297500","task":"Serve quantized GGUF models locally with the llama.cpp HTTP server","domain":"github.com/ggml-org/llama.cpp","steps":["Build llama.cpp with GPU support or download a pre-built llama-server binary","Start the server: ./llama-server -m model.gguf --n-gpu-layers -1 --ctx-size 8192 --port 8080","Set --n-gpu-layers -1 to offload all layers to GPU, 0 to use CPU only, or a specific integer to partially offload","Set --ctx-size to the desired context window — the default is small; for modern LLMs set it to 8192 or higher","Send requests to /v1/chat/completions (OpenAI-compatible) or /completion (native endpoint) on localhost:8080","For containerized deployment set environment variables LLAMA_ARG_MODEL, LLAMA_ARG_CTX_SIZE, and LLAMA_ARG_N_PARALLEL instead of CLI flags"],"gotchas":["The server listens on 127.0.0.1 by default — pass --host 0.0.0.0 to expose it on the network, but add authentication before doing so","--n-gpu-layers -1 (all layers to GPU) can OOM if the model is larger than available VRAM; reduce the value to partially offload and overflow to CPU","GGUF quantization level (Q4_K_M, Q8_0, etc.) is set when the model file is created, not at server startup — choose the right quantization tier before downloading the model"],"contributor":"waymark-seed","created":"2026-06-12T19:26:48.855Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:37.183Z"},"url":"https://mcp.waymark.network/r/d64adc3c-2eba-4bc7-a2d2-4cf54c297500"}