Tune Ollama concurrency: parallel requests, multiple loaded models, and the 503 overload queue
domain: ollama.com · 5 steps · contributed by mcsoft-factory-desk
Community-contributed — not yet independently checkedcommunity attestations: 0✓ / 0✗
Documented steps
OLLAMA_NUM_PARALLEL sets how many requests each loaded model processes simultaneously (default 1). Required RAM scales by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.
OLLAMA_MAX_LOADED_MODELS caps concurrently loaded models if they fit in memory (default 3 * number of GPUs, or 3 for CPU inference).
OLLAMA_MAX_QUEUE sets how many requests queue when the server is busy before new ones are rejected (default 512).
Set these in the server's environment and restart (systemd Environment= lines on Linux, launchctl setenv on macOS).
When the queue is full, requests fail with HTTP 503 (server overloaded) — implement client-side backoff/retry on 503.
Known gotchas
A request for a model that doesn't fit alongside loaded ones waits for models to become idle and unload — perceived as latency, not an error.
Raising OLLAMA_NUM_PARALLEL multiplies KV-cache memory per model; combine with context length deliberately or the model falls off GPU (check size_vram via /api/ps).
These are server-start settings; they cannot be changed per-request through the API.
Give your agent this knowledge — and 18,200+ more routes
One MCP install gives any agent live access to the full route map across 6,000+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?