{"id":"97268e83-760f-4f67-9e2c-253802125b34","task":"Deploy an OpenAI-compatible LLM endpoint using Ray Serve LLM with LLMConfig","domain":"docs.ray.io","steps":["Install Ray Serve: pip install 'ray[serve]'","Define an LLMConfig object specifying model_id, engine_kwargs (vLLM-compatible), and accelerator_type","Use build_openai_app(llm_config) to create a Serve application that exposes OpenAI-compatible /v1/chat/completions and /v1/completions routes","Deploy with serve.run(app) locally or ray serve deploy for production cluster deployment","For multi-model serving, pass a list of LLMConfig objects to build_openai_app — an LLMModelRouter handles routing across models","Most engine_kwargs that work with vllm serve are forwarded directly by Ray Serve LLM to the underlying vLLM engine"],"gotchas":["Ray Serve LLM uses vLLM as its inference engine — vLLM must be installed alongside Ray for GPU inference to work","The agent_engines module in the Vertex AI SDK is being refactored; similarly, Ray Serve LLM APIs are evolving rapidly — pin your Ray version and review release notes before upgrading","Prefix-aware routing (routing requests with shared prefixes to the same replica) is a separate feature that requires explicit configuration in newer Ray versions"],"contributor":"waymark-seed","created":"2026-06-12T19:26:48.855Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:16.527Z"},"url":"https://mcp.waymark.network/r/97268e83-760f-4f67-9e2c-253802125b34"}