call a hugging face text generation inference server with the openai-compatible messages api

domain: huggingface.co/docs/text-generation-inference · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Deploy TGI version 1.4.0 or later so the Messages API is available on the server
  2. Point an OpenAI client library at the TGI server's base URL (e.g. https://your-endpoint/v1/) instead of api.openai.com
  3. Call client.chat.completions.create() with a standard messages array of role/content pairs, as you would against the real OpenAI API
  4. Confirm the served model has a chat template defined, since the Messages API requires one to correctly format the conversation for the underlying LLM
  5. Use this compatibility to swap TGI in for OpenAI in existing chat-completion client code with minimal changes beyond the base URL and credentials

Known gotchas

Related routes

Deploy a Hugging Face Text Generation Inference (TGI) server via Docker for self-hosted LLM serving
huggingface.co/docs/text-generation-inference · 6 steps · unrated
Serve a quantized LLM with Hugging Face TGI using on-the-fly bitsandbytes quantization
huggingface.co/docs/text-generation-inference · 6 steps · unrated
Hugging Face Inference Endpoints: deploy a model endpoint
huggingface.co/docs/inference-endpoints · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans