Portkey AI Gateway Config

Portkey is an AI gateway that sits between your application and LLM providers. It provides a unified API (compatible with OpenAI's SDK), automatic fallbacks, load balancing, request caching, observability, and cost tracking. Virtual keys let you manage multiple provider API keys centrally without exposing them in your code.

Fallback tries providers in order — it only moves to the next provider if the current one fails (based on status codes like 429 or 503). Load Balance distributes requests across providers based on weights — useful for cost optimization (e.g. 70% cheap model, 30% expensive model). Use Fallback for reliability, Load Balance for cost optimization.

Virtual keys are Portkey's abstraction over real provider API keys. You add your actual OpenAI/Anthropic keys to Portkey's vault, and Portkey gives you a virtual key (starting with portkey-vk-...). You use virtual keys in your config instead of real keys, so you can rotate/revoke real keys without changing your application code.

Yes. Portkey is designed as a drop-in replacement. Just change `client = OpenAI(api_key=...)` to `client = Portkey(api_key=..., virtual_key=...)`. The rest of your code stays identical. The same works for Node.js with the OpenAI Node SDK.

Simple caching does exact string matching — the same prompt returns the same cached response instantly. Semantic caching uses vector embeddings to match 'similar' prompts, so 'What's 2+2?' and 'What is 2 plus 2?' return the same cached result. Simple caching is great for API wrappers with predictable prompts. Semantic caching works well for chatbots and QA systems with paraphrased queries.