Serve quantized GGUF models locally with the llama.cpp HTTP server

domain: github.com/ggml-org/llama.cpp · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Build llama.cpp with GPU support or download a pre-built llama-server binary
  2. Start the server: ./llama-server -m model.gguf --n-gpu-layers -1 --ctx-size 8192 --port 8080
  3. Set --n-gpu-layers -1 to offload all layers to GPU, 0 to use CPU only, or a specific integer to partially offload
  4. Set --ctx-size to the desired context window — the default is small; for modern LLMs set it to 8192 or higher
  5. Send requests to /v1/chat/completions (OpenAI-compatible) or /completion (native endpoint) on localhost:8080
  6. For containerized deployment set environment variables LLAMA_ARG_MODEL, LLAMA_ARG_CTX_SIZE, and LLAMA_ARG_N_PARALLEL instead of CLI flags

Known gotchas

Related routes

configure llama.cpp server continuous batching and parallel request slots
github.com/ggml-org/llama.cpp · 5 steps · unrated
constrain llama.cpp server output to a schema using gbnf grammars
github.com/ggml-org/llama.cpp · 5 steps · unrated
Serve a quantized LLM with Hugging Face TGI using on-the-fly bitsandbytes quantization
huggingface.co/docs/text-generation-inference · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans