LiteLLM Proxy: Per-Key Budgets and Fallbacks (v1.110.0)
Weekend spend cannot be attributed to a feature, a provider outage takes the app down instead of failing over, and the fallback list looks configured but never fires.
📜 Table of Contents
- What "verified" means here
- File 1: the compose stack
- File 2: config.yaml with an alias in front of two providers
- The fallback mistake: a list of strings
- File 3: one key per feature, with a cap
- File 4: the two ways to see the money
- Per-call cost, so the caps mean something
- Checklist before you point real traffic at it
The failure this setup prevents is not an outage. It is the Monday morning question: which feature burned $180 over the weekend, and on whose key? LiteLLM is a proxy that sits between your app and every model provider, and it answers that question only if you configure three things: a database, a master key, and one virtual key per feature with a budget on it. The fallback list is the fourth piece, and it is the one people get wrong.
What "verified" means here
Everything below was checked against the source of v1.110.0, released 18 November 2026 (GitHub releases API), rather than copied from a blog. Config fields carry the file and line they live in, so a reader on a different version can confirm or refute each one. The compose image and port come from the repository's own docker-compose.yml at that tag. This is a configuration walkthrough, not a load test: nothing here reports throughput or latency numbers, because those were not measured.
File 1: the compose stack
The proxy needs Postgres if you want keys, budgets and spend logs to survive a restart. The upstream docker-compose.yml at v1.110.0 runs two services, litellm on 4000:4000 using image docker.litellm.ai/berriai/litellm:main-stable, and a db service. It sets DATABASE_URL and STORE_MODEL_IN_DB=True. A trimmed version that keeps the same wiring:
services:
db:
image: postgres:16
environment:
POSTGRES_USER: llmproxy
POSTGRES_PASSWORD: change-me
POSTGRES_DB: litellm
volumes:
- pgdata:/var/lib/postgresql/data
litellm:
image: docker.litellm.ai/berriai/litellm:main-stable
ports: ["4000:4000"]
volumes:
- ./config.yaml:/app/config.yaml
command: ["--config", "/app/config.yaml"]
environment:
DATABASE_URL: postgresql://llmproxy:change-me@db:5432/litellm
STORE_MODEL_IN_DB: "True"
LITELLM_MASTER_KEY: sk-master-change-me
DEEPSEEK_API_KEY: ${DEEPSEEK_API_KEY}
OPENAI_API_KEY: ${OPENAI_API_KEY}
depends_on: [db]
volumes:
pgdata:
Two notes that cost time if you skip them. STORE_MODEL_IN_DB is what lets the model list be edited after boot; without it the proxy still serves config.yaml models but the admin UI has nothing to write to. And general_settings.master_key is optional in a way that matters: the docs state that if it is set, every call to the proxy must carry either that key or a generated one - leave it unset and the proxy will answer unauthenticated requests.
File 2: config.yaml with an alias in front of two providers
The point of the alias is that your application code never names a provider. It asks for chat-default; the proxy decides who serves it.
model_list:
- model_name: chat-default
litellm_params:
model: deepseek/deepseek-flash
api_key: os.environ/DEEPSEEK_API_KEY
- model_name: chat-default
litellm_params:
model: openai/gpt-5-mini
api_key: os.environ/OPENAI_API_KEY
- model_name: embed-default
litellm_params:
model: openai/text-embedding-3-small
api_key: os.environ/OPENAI_API_KEY
router_settings:
num_retries: 2
timeout: 45
allowed_fails: 3
cooldown_time: 30
fallbacks: [{"chat-default": ["chat-backup"]}]
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
litellm_settings:
drop_params: true
Field names and their shapes, from the tagged source: fallbacks is list[dict] (litellm/types/router.py:230), and so are context_window_fallbacks and content_policy_fallbacks; num_retries, timeout, allowed_fails and cooldown_time sit on the same settings object (router.py:224-227).
The fallback mistake: a list of strings
Most snippets floating around write fallbacks: ["chat-backup"]. In v1.110.0 the field is a list of dictionaries keyed by the primary model name. A list of strings does not express "when chat-default fails, use chat-backup" - there is no edge to follow. Write the dict. And if chat-backup is not defined in model_list, the edge points at a model that does not exist, which shows up as a failed request rather than a startup error. Grep your config for the backup name before you deploy.
Three different failures need three different lists. A 500 or a timeout is handled by fallbacks. A prompt longer than the primary model's window is context_window_fallbacks. A provider-side content policy block is content_policy_fallbacks. Set only the first and a 200k-token prompt will keep failing against the small model forever, retrying.
File 3: one key per feature, with a cap
A key carries the budget, not the config file. The key endpoints live in litellm/proxy/management_endpoints/key_management_endpoints.py: /key/generate (line 2347), /key/info (5030), /v2/key/info, /key/update (4019), /key/delete (4753). Budget fields on the key object include max_budget, budget_duration and model_max_budget (litellm/proxy/_types.py:1755-1830), the last one with the documented shape {'gpt-4o': {'max_budget': ..., 'budget_duration': '1d', 'tpm_limit': 1000, 'rpm_limit': 1000}}.
# key.json
{
"key_alias": "nightly-summaries",
"models": ["chat-default", "embed-default"],
"max_budget": 50,
"budget_duration": "30d",
"rpm_limit": 120
}
curl -s http://localhost:4000/key/generate \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d @key.json
The response contains the key itself, prefixed sk-. Give the nightly job that key and nothing else. When it hits $50 in 30 days the proxy answers 400 with a budget-exceeded error instead of quietly continuing. /key/info?key=sk-... returns spend, budget, remaining and reset time, which is the number to put on a dashboard.
The key schema also carries a budget_fallbacks field (_types.py:1274), so a budget-driven redirect is configured per key rather than globally in router_settings. That is as far as the reading goes: the field exists, its runtime behaviour was not exercised here, so treat it as something to test on a staging key rather than trust from this page.
File 4: the two ways to see the money
Live counters: /key/info per key, /spend/keys and /spend/users for aggregates. Raw rows: /spend/logs, and the source warns that this call truncates its result to the most recent rows and points at /spend/logs/v2 for pagination (litellm/proxy/spend_tracking/spend_management_endpoints.py:380). For a month-end reconciliation, pull v2 and sum, or query Postgres directly; the truncated endpoint will silently undercount and make a busy week look cheap.
Provider reachability is separate: /health (health_endpoints.py:1557) and /v1/models (proxy_server.py:12774) both need a key, so use the master key when you wire them into monitoring. A fallback that is firing constantly looks like a healthy service from the app's side, and only the health endpoint shows the primary sitting in cooldown.
Per-call cost, so the caps mean something
A $50 monthly cap is arbitrary until you know what a call costs. Take a retrieval prompt of 2,198 tokens in and a 350-token answer, the shape measured in our RAG cost breakdown: at DeepSeek's published off-peak rate for deepseek-flash ($0.15 per million input, $0.60 per million output) that call is $0.00054; at the gpt-5-mini rate carried in LiteLLM's own price map ($0.25 / $2.00) it is $0.00125, 2.3 times as much. So the same $50 buys roughly 92,000 calls on the first and 40,000 on the second. That is the number to put next to the key, not the dollar figure.
One more lever the proxy makes visible: a model entry with a stable instruction prefix benefits from provider-side prompt caching, and DeepSeek prices cached input at $0.003 per million against $0.15 cache-miss, fifty times cheaper. Routing through an alias does not create the cache, but it does make the switch to a cache-friendly provider a one-line change.
Checklist before you point real traffic at it
One: docker compose config parses; the volume path for config.yaml points at the file, not the directory. Two: /v1/models with the master key lists every model_name you expect, including backups. Three: break one provider's credentials and confirm the request still returns 200 - that is the only proof the fallback edge resolves. Four: generate a key with max_budget 0.01, spend two cents, and watch the 400 arrive. Five: pull /spend/logs/v2 and confirm the row carries the key alias, not just the model.
Related reading on this site: the gateway comparison covers why a proxy beats per-SDK keys, and the KV cache explainer covers the memory side of serving your own model behind the same alias.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.