{"id":"9c1b96b2-671b-4ff0-93ed-933f15d1c6d5","task":"Ray Serve: configure autoscaling for a deployment (min_replicas, max_replicas, target_ongoing_requests)","domain":"ml-ops","steps":["Decorate your deployment with @serve.deployment(...) and either set num_replicas=\"auto\" to use Ray Serve's autoscaling defaults, or explicitly pass autoscaling_config={\"min_replicas\": 1, \"max_replicas\": 10, \"target_ongoing_requests\": 2} to control it yourself.","Separately set max_ongoing_requests on the deployment (a top-level option, not part of autoscaling_config) to a value above target_ongoing_requests so Serve can queue requests while it scales.","Deploy the app with `serve run` (or a YAML config listing `num_replicas: auto` under the deployment); the Serve autoscaler polls each replica's queue and adds/removes replicas to keep ongoing requests near target_ongoing_requests.","Set min_replicas=0 if traffic has long idle windows and you can tolerate extra latency on the first request while a replica cold-starts; otherwise size it for your low-traffic floor.","Set max_replicas to roughly 20% above your estimated peak-traffic replica need."],"gotchas":["If you configure autoscaling_config manually without num_replicas=\"auto\", the base default for max_replicas is 1 — the deployment will silently never scale beyond 1 replica unless you raise it explicitly.","num_replicas=\"auto\" is shorthand for target_ongoing_requests=2, min_replicas=1, max_replicas=100, and max_ongoing_requests=5 — you can override any of these by supplying your own autoscaling_config alongside num_replicas=\"auto\".","min_replicas=0 saves resources during idle periods but adds tail latency to the first request after idle while a new replica starts."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/9c1b96b2-671b-4ff0-93ed-933f15d1c6d5"}