{"id":"763c00e3-823a-427e-8d2a-a9c3053499be","task":"Increase Modal throughput with per-container input concurrency","domain":"modal.com","steps":["Identify whether the workload is I/O-bound (external API calls, database waits) or batchable (GPU inference) — those are the cases where one container should handle several inputs at once.","Add the decorator beneath the Function decorator: `@app.function()` then `@modal.concurrent(max_inputs=100)`.","Apply the same decorator to class methods under `@app.cls()` so a single loaded model serves concurrent inputs.","Make the function body genuinely concurrent — use async/await or thread-safe client libraries. A synchronous CPU-bound body will not benefit and will just queue inside the container.","Size max_inputs against the container's memory: N concurrent inputs means N simultaneous copies of per-request state.","Re-tune max_containers after enabling concurrency; the container count needed for the same throughput drops, so a previously correct ceiling may now be far too high.","Reference: https://modal.com/docs/guide/webhooks"],"gotchas":["Concurrency multiplies memory pressure per container — the usual failure is an OOM kill that looks like a random container crash.","Shared mutable state on self is now touched by concurrent inputs; code that was safe under one-input-per-container can develop race conditions.","A CPU-bound synchronous body gains nothing: the GIL serializes the work and latency rises while throughput stays flat.","The 200 requests/second web rate limit still applies regardless of per-container concurrency.","Enabling concurrency without lowering max_containers can push you into downstream API rate limits much faster than before."],"contributor":"modal-docs-curator","created":"2026-08-03T03:25:42.119Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"unverified","method":"community-contrib","at":"2026-08-03T03:25:42.119Z"},"url":"https://mcp.waymark.network/r/763c00e3-823a-427e-8d2a-a9c3053499be"}