LiteLLM Proxy Config Generator

LiteLLM is an open-source library that provides a unified OpenAI-compatible API across 100+ LLM providers (Anthropic, Gemini, Groq, Cohere, etc.). Running it as a proxy means your apps always call one endpoint — the proxy handles routing, fallbacks, retries, and budget tracking. You can swap models without changing application code.

LiteLLM supports 4 routing strategies: (1) Simple Shuffle — weighted random selection across providers, (2) Least Busy — route to the provider with fewest active requests, (3) Latency-based — tracks P95 latency and routes to the fastest provider, (4) Cost-based — estimates token cost per provider and routes to the cheapest. Latency-based is best for production latency-sensitive apps.

Use os.environ/KEY_NAME syntax in config.yaml — LiteLLM reads from environment variables at runtime. Never hardcode API keys in config files. For Docker deployments, pass keys via --env-file or Kubernetes secrets. The master_key in general_settings is used to authenticate requests to your proxy itself.

budget_duration combined with max_budget defines a spending limit per period. Setting max_budget: 100 with budget_duration: 30d means the proxy will refuse requests after $100 is spent in any 30-day rolling window. You can track spending in the LiteLLM dashboard UI (port 4000 by default) or via /budget/info API.

Yes for high-volume, low-variance requests. LiteLLM's Redis caching deduplicates identical prompts — if 100 users ask the same question, only one LLM call is made. Use a TTL of 1-24 hours for stable content. Avoid caching for personalized, real-time, or safety-critical responses. Make sure your Redis instance is in the same region as your proxy for minimal latency.