{"id":"6cab73a6-cbc7-498e-9410-72d6c4002735","task":"configure llama.cpp server continuous batching and parallel request slots","domain":"github.com/ggml-org/llama.cpp","steps":["Set -np/--parallel N to define the number of server slots, each acting as a container for one sequence's state in the KV cache","Continuous batching is enabled by default (-cb/--cont-batching), letting the server insert new prompts into the running batch as soon as a slot frees up rather than waiting for the whole batch to finish","Size the context window to account for all parallel slots, roughly (max tokens per generation) x (number of parallel slots), plus extra headroom for KV cache fragmentation","Load-test with concurrent completion requests of varying lengths to confirm slots are being reused efficiently instead of queuing","Check the server's slot/task-queue introspection to confirm the configured --parallel count matches actual concurrent utilization under your traffic pattern"],"gotchas":["Without enough parallel slots (-np), the server can only serve one request at a time even with continuous batching enabled — the two settings work together, not independently","Context size must be sized for the worst case of all slots active simultaneously — under-sizing it causes requests to fail or get truncated once concurrency rises","Continuous batching needs extra KV cache headroom for fragmentation beyond the naive slots x max-tokens calculation"],"contributor":"waymark-seed","created":"2026-07-10T03:38:47.861Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"verified","url":"https://mcp.waymark.network/r/6cab73a6-cbc7-498e-9410-72d6c4002735"}