LLM Serving With vLLM 2026: Deploying Open Models as a Production API。Once the model is chosen, the questions become engineering ones: serve dozens of users, keep latency stable, e

📜 Table of Contents

• Why it is fast
• Minimal deployment
• Key flags
• Throughput versus latency
• Production checklist
• When not to use vLLM
• Ramp-up order
• FAQ
• Related

vLLM Inference Service 2026: Turning an Open Model into a Production API

Once the model is chosen, the questions become engineering ones: serve dozens of users, keep latency stable, expose an OpenAI-compatible API, and debug what breaks. vLLM is the mainstream open-source answer in 2026.

Why it is fast

PagedAttention allocates the KV cache in fixed-size blocks on demand, like virtual memory paging, which cuts fragmentation and fits more concurrent requests and longer contexts. Continuous batching starts new requests as slots free up instead of waiting for a full batch, keeping the GPU busy. Together they multiply throughput over naive per-request serving; the exact multiple depends on model size, sequence length and traffic, so benchmark your load.

Minimal deployment

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 \
  --max-model-len 8192 --gpu-memory-utilization 0.90

The endpoint is OpenAI compatible, so client code only changes base_url. Containers use vllm/vllm-openai with the serve command as arguments; mount the model cache or every restart re-downloads weights.

Key flags

Rough VRAM math: parameters times bytes per parameter plus KV cache. A 7B FP16 model is about 14GB of weights, so a 16GB card needs quantization or a shorter context to be comfortable.

Throughput versus latency

More sequences per batch raises throughput and p95 latency; lower concurrency, speculative decoding and shorter contexts cut latency. For long documents raise context and lower concurrency; for multi-turn chat enable prefix caching. Load test and read p50 and p95, not the mean. Pushing max-num-seqs higher past the point where KV cache saturates makes throughput fall, so find the knee instead of maxing the flag.

Production checklist

Prometheus metrics for queue depth, KV cache usage and running requests; autoscaling on GPU utilization or queue length rather than CPU; a load balancer aware of long request timeouts; rate limits at the gateway; request and token logging for cost; rolling restarts for model upgrades.

When not to use vLLM

Small cards (8GB) and 7B models, CPU-only hosts (use llama.cpp server), single-user local work (Ollama is simpler), and cases that need a GUI. At extreme throughput, SGLang is worth benchmarking.

Ramp-up order

Run with defaults and verify with curl on day one, connect the real application on day two, then tune one parameter per iteration while recording metrics. Changing five flags at once makes regressions unattributable.

FAQ

Out of memory at startup? Lower --gpu-memory-utilization, shorten --max-model-len, use quantized weights, reduce --max-num-seqs. If it still fails, the card is too small for that model.

Multiple models? One instance per model behind a router keyed on model name.

Does quantization hurt? Slight for general tasks, more visible in long reasoning and code generation. Compare on your own test set.

Tool calling? Supported with per-model chat template and tool parser flags; check the official docs for exact names.

Related

• LLM inference optimization
• Ollama vs llama.cpp
• FastChat serving guide