Configure the Portkey AI Gateway with fallbacks, retries, weighted load balancing, and conditional routing across LLM providers

Documented steps

1. Install the SDK: pip install portkey-ai (import portkey_ai) or npm install portkey-ai. The REST base URL is https://api.portkey.ai/v1 if you call it directly.
2. Create a Portkey API key in the dashboard and store it as PORTKEY_API_KEY.
3. Set up providers under the current Model Catalog system: create an Integration at the org level to hold credentials, then reference it at workspace level as an AI Provider. This is what used to be called a Virtual Key; existing Virtual Keys were auto-migrated and kept their slugs.
4. Decide how to reference providers. The new recommended syntax is model="@provider_slug/model_slug". The legacy virtual_key config field and x-portkey-virtual-key header still work, and Portkey states existing code, configs, and prompts continue to work unchanged.
5. Build a Config JSON object. Documented top-level fields include strategy, provider, api_key, virtual_key (legacy), targets, cache (mode simple or semantic, max_age), retry, weight, on_status_codes, override_params, default_params, drop_params, request_timeout, custom_host, forward_headers, passthrough, cb_config (circuit breaker), name, strict_open_ai_compliance, plus cloud-specific fields and guardrail hooks.
6. For fallbacks, set strategy.mode to "fallback" and list targets in priority order: {"strategy": {"mode": "fallback", "on_status_codes": [429, 503]}, "targets": [{"provider": "@provider-slug"}, {"override_params": {"model": "@provider-slug-2/model-name"}}]}. Omitting on_status_codes falls over on any non-2xx.
7. For weighted load balancing, set strategy.mode to "loadbalance" and give each target a weight, for example targets with weight 70 and 30. Weights are relative and Portkey normalizes them to 100 percent; the default weight is 1.
8. For session affinity, add sticky to the load-balance strategy: {"sticky": {"enabled": true, "hash_fields": ["metadata.session_id"], "ttl": 3600}}. The default ttl is 3600 seconds.
9. For conditional routing, set strategy.mode to "conditional" with an ordered conditions array of {query, then} pairs and a default target, and give each entry in targets a name to reference. Query paths support metadata.<key>, params.<key>, and url.pathname. Operators are $eq, $ne, $in, $nin, $regex, $gt, $gte, $lt, $lte, $and, $or.
10. Add retries at the top level or per target: {"retry": {"attempts": 3, "on_status_codes": [429, 503], "use_retry_after_headers": true}}. Maximum attempts is 5.
11. Save the Config in the dashboard to get a Config ID of the form pc-***, or pass the raw JSON inline.
12. Attach the Config to requests either via the SDK constructor (Portkey({ apiKey, config: 'pc-***' })) or via the x-portkey-config header, which accepts either a JSON object or the config ID string. Other request headers are x-portkey-api-key, x-portkey-provider, x-portkey-virtual-key (legacy), x-portkey-custom-host, and Authorization: Bearer <provider key> where needed.
13. Test each routing branch before rollout: force a provider failure to confirm fallback fires, send N requests to observe the weighted distribution, and exercise every conditional branch including the default.
14. Monitor the x-portkey-retry-attempt-count response header, where -1 means failed, 0 means no retry was needed, and a positive number is how many retries it took to succeed. Confirm costs roll up under the correct AI Provider.