AI Gateway 2026: LiteLLM vs Portkey - Route, Cache, and Monitor Every LLM Call (55k Stars)

You have three LLM providers and one of them goes down at the worst moment. An AI gateway routes around it, tracks cost, and keeps one API for your whole team.

## AI Gateways: The Load Balancer for Language Models An AI gateway sits between your application and every LLM provider, exposing one unified API (usually OpenAI-compatible) while handling routing, retries, caching, cost tracking, and key management. It is the piece of infrastructure you do not need until you have two providers - and cannot live without once you have three. The two dominant open-source options are LiteLLM (55,260 GitHub stars, August 2026) and Portkey (12,622 stars). ## LiteLLM: The Lightweight Standard LiteLLM is a Python proxy that speaks the OpenAI API format and translates it to 100+ providers - OpenAI, Anthropic, Gemini, Bedrock, Azure, and local vLLM or Ollama endpoints. You run it as a Docker container, point your app at it, and change providers by editing a config file. It also gives you cost tracking per request, rate limiting, and fallbacks (if provider A fails, retry on provider B). ## Portkey: The Enterprise-Heavy Gateway Portkey offers the same core (unified API, routing, fallbacks) plus deeper guardrails, prompt versioning, and an observability dashboard with traces. It is a good fit when you need auditability and team governance - who changed which prompt, and what did it cost - out of the box. ## What You Actually Get With Either - **One API key for the team** - no more sharing provider keys across services. - **Automatic failover** - route to a backup provider when the primary is rate-limited or down. - **Cost tracking** - per-request spend across all providers in one dashboard. - **Caching** - identical requests hit a cache instead of the API, cutting cost and latency. - **Retries with backoff** - handle transient 429s and 5xx without app code. ## The Architecture That Works Run the gateway as a sidecar service next to your app. Point all internal clients at it. Keep the upstream keys in the gateway's environment, never in app code. This also gives you a choke point where you can add a budget - when spend hits $X this month, the gateway starts rejecting non-critical calls. ## FAQ **Is LiteLLM free?** Yes - MIT-licensed open source; the company offers a hosted enterprise version. **Do I need a gateway for one provider?** No. Add it when you have two or more providers or need team-wide cost control. **Does it work with local models?** Yes - vLLM, Ollama, and any OpenAI-compatible local endpoint work as upstreams. **Gateway vs just switching providers in code?** A gateway makes switching a config change instead of a code change - and adds failover you cannot easily build by hand.