{"id":"5769cc08-a37a-429c-9c81-4fdcbf3e41aa","task":"MLflow Deployments Server (AI Gateway): stand up a gateway server to proxy a third-party LLM provider endpoint","domain":"ml-ops","steps":["Install MLflow with GenAI extras and start the tracking server: pip install 'mlflow[genai]' then mlflow server --port 5000. The AI Gateway is built into the MLflow Tracking Server itself and comes up at http://localhost:5000 (no separate gateway process needed).","Note the backend requirement: the gateway needs a SQL-based backend store (SQLite, PostgreSQL, MySQL, or MSSQL) plus the FastAPI tracking server; the default `mlflow server` setup (SQLite + FastAPI) already satisfies this for local use.","In the MLflow UI, go to Settings > LLM Connections > Create, name the connection, pick the provider (e.g. OpenAI), and paste that provider's API key — MLflow stores it encrypted server-side.","Go to the Endpoints tab > Create Endpoint, give it a name, pick the provider/model (e.g. gpt-4o), and attach the LLM Connection you just created.","Query the endpoint either via a plain POST to http://localhost:5000/gateway/<endpoint-name>/mlflow/invocations with a {\"messages\": [...]} body, or by pointing an OpenAI-SDK client's base_url at http://localhost:5000/gateway/mlflow/v1 (no client-side API key needed since it's configured server-side).","Use the gateway's traffic-routing features (documented under Traffic Routing & Fallbacks) to configure A/B splits across models/providers or automatic failover chains for high availability."],"gotchas":["The gateway is served from the same process/port as the MLflow Tracking Server (default 5000) — it is not a standalone binary you deploy separately.","You must use one of the supported SQL backend stores (SQLite/PostgreSQL/MySQL/MSSQL) with the FastAPI tracking server; other tracking-store configurations won't expose the gateway.","Provider API keys are entered and stored through LLM Connections in the UI/API rather than passed by the calling client, so client code never needs the real provider key."],"contributor":"waymark-seed","created":"2026-07-09T00:09:27Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/5769cc08-a37a-429c-9c81-4fdcbf3e41aa"}