Verified steps

Build llama.cpp with GPU support or download a pre-built llama-server binary
Start the server: ./llama-server -m model.gguf --n-gpu-layers -1 --ctx-size 8192 --port 8080
Set --n-gpu-layers -1 to offload all layers to GPU, 0 to use CPU only, or a specific integer to partially offload
Set --ctx-size to the desired context window — the default is small; for modern LLMs set it to 8192 or higher
Send requests to /v1/chat/completions (OpenAI-compatible) or /completion (native endpoint) on localhost:8080
For containerized deployment set environment variables LLAMA_ARG_MODEL, LLAMA_ARG_CTX_SIZE, and LLAMA_ARG_N_PARALLEL instead of CLI flags

Known gotchas

The server listens on 127.0.0.1 by default — pass --host 0.0.0.0 to expose it on the network, but add authentication before doing so
--n-gpu-layers -1 (all layers to GPU) can OOM if the model is larger than available VRAM; reduce the value to partially offload and overflow to CPU
GGUF quantization level (Q4_K_M, Q8_0, etc.) is set when the model file is created, not at server startup — choose the right quantization tier before downloading the model

huggingface.co/docs/text-generation-inference · 6 steps · unrated

Serve an LLM with vLLM using tensor parallelism across multiple GPUs

docs.vllm.ai · 6 steps · unrated

Configure low-latency CMAF chunked fMP4 live packaging with FFmpeg and a simple origin

ffmpeg.org · 6 steps · unrated

Give your agent this knowledge — and 200+ more routes

One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Serve quantized GGUF models locally with the llama.cpp HTTP server

Verified steps

Known gotchas

Related routes

Give your agent this knowledge — and 200+ more routes