AI Gateway 2026：LiteLLM vs Portkey，统一路由、缓存与监控
你有三家LLM供应商，其中一家在最糟的时刻宕机。AI网关负责绕路、记成本、统一API。
AI Gateways: The Load Balancer for Language Models
An AI gateway sits between your application and every LLM provider, exposing one unified API (usually OpenAI-compatible) while handling routing, retries, caching, cost tracking, and key management. It is the piece of infrastructure you do not need until you have two providers - and cannot live without once you have three. The two dominant open-source options are LiteLLM (55,260 GitHub stars, August 2026) and Portkey (12,622 stars).
LiteLLM: The Lightweight Standard
LiteLLM is a Python proxy that speaks the OpenAI API format and translates it to 100+ providers - OpenAI, Anthropic, Gemini, Bedrock, Azure, and local vLLM or Ollama endpoints. You run it as a Docker container, point your app at it, and change providers by editing a config file. It also gives you cost tracking per request, rate limiting, and fallbacks (if provider A fails, retry on provider B).
Portkey: The Enterprise-Heavy Gateway
Portkey offers the same core (unified API, routing, fallbacks) plus deeper guardrails, prompt versioning, and an observability dashboard with traces. It is a good fit when you need auditability and team governance - who changed which prompt, and what did it cost - out of the box.
What You Actually Get With Either
- One API key for the team - no more sharing provider keys across services.
- Automatic failover - route to a backup provider when the primary is rate-limited or down.
- Cost tracking - per-request spend across all providers in one dashboard.
- Caching - identical requests hit a cache instead of the API, cutting cost and latency.
- Retries with backoff - handle transient 429s and 5xx without app code.
The Architecture That Works
Run the gateway as a sidecar service next to your app. Point all internal clients at it. Keep the upstream keys in the gateway's environment, never in app code. This also gives you a choke point where you can add a budget - when spend hits $X this month, the gateway starts rejecting non-critical calls.
FAQ
Is LiteLLM free? Yes - MIT-licensed open source; the company offers a hosted enterprise version.
Do I need a gateway for one provider? No. Add it when you have two or more providers or need team-wide cost control.
Does it work with local models? Yes - vLLM, Ollama, and any OpenAI-compatible local endpoint work as upstreams.
Gateway vs just switching providers in code? A gateway makes switching a config change instead of a code change - and adds failover you cannot easily build by hand.
