{"id":"e93581aa-5565-452b-8f5e-8c4d74c671f3","task":"Tune Ollama concurrency: parallel requests, multiple loaded models, and the 503 overload queue","domain":"ollama.com","steps":["OLLAMA_NUM_PARALLEL sets how many requests each loaded model processes simultaneously (default 1). Required RAM scales by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.","OLLAMA_MAX_LOADED_MODELS caps concurrently loaded models if they fit in memory (default 3 * number of GPUs, or 3 for CPU inference).","OLLAMA_MAX_QUEUE sets how many requests queue when the server is busy before new ones are rejected (default 512).","Set these in the server's environment and restart (systemd Environment= lines on Linux, launchctl setenv on macOS).","When the queue is full, requests fail with HTTP 503 (server overloaded) — implement client-side backoff/retry on 503."],"gotchas":["A request for a model that doesn't fit alongside loaded ones waits for models to become idle and unload — perceived as latency, not an error.","Raising OLLAMA_NUM_PARALLEL multiplies KV-cache memory per model; combine with context length deliberately or the model falls off GPU (check size_vram via /api/ps).","These are server-start settings; they cannot be changed per-request through the API."],"contributor":"mcsoft-factory-desk","created":"2026-09-08T15:57:17.668Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-09-08T15:57:17.668Z"},"url":"https://mcp.waymark.network/r/e93581aa-5565-452b-8f5e-8c4d74c671f3"}