LLM Inference Optimization 2026
Your self-hosted model serves two tokens per second while the GPU sits half idle. The big wins cluster in a few well-understood techniques - start by finding the real bottleneck.
📜 Table of Contents
LLM Inference Optimization 2026: Batching, KV Cache and Speculative Decoding Explained
LLM inference is memory-bound, not compute-bound. Every generated token streams the entire weight set from VRAM once. That single fact explains every optimization below: they all reduce memory traffic or reuse data already loaded.
Track two metrics: TTFT (time to first token) and generation speed (tokens per second). Users feel those, not aggregate throughput. Measure before optimizing or you will tune the wrong thing.
Two workloads, two directions
Single-user interactive: latency-bound, one request at a time, low GPU utilization is normal. Wins come from cutting per-step cost: quantization, KV cache, speculative decoding.
Multi-user serving: throughput-bound; an idle GPU is wasted money. Wins come from packing requests into the same forward pass.
Most "optimization did nothing" stories are batching techniques applied to single-user latency, or the reverse.
Technique 1: continuous batching
Naive servers handle one request at a time and wait for memory between steps. Continuous batching interleaves tokens from many requests into one batch and refills slots as requests finish. vLLM's PagedAttention manages KV cache in pages so it need not reserve max-length contiguous memory per request; SGLang pushes scheduling and prefix reuse further. Against naive serving these often deliver several-fold throughput gains, varying widely with model and concurrency.
Cost: slightly higher per-request latency, since batches wait to fill. Gains scale with concurrency; a single user sees almost none.
Technique 2: KV cache and its quantization
Attention looks at all previous tokens for each new one. Recomputing that every step makes cost grow linearly with length, so engines cache keys and values - at the price of VRAM.
Roughly: KV cache ≈ 2 × layers × KV heads × head dim × sequence length × bytes per element. For an 8B-class model at 8K context, that is often several GB at FP16 - and with long contexts the cache can exceed the weights.
./build/bin/llama-cli -m model.gguf -c 8192 -ctk q8_0 -ctv q8_0
vllm serve model --kv-cache-dtype fp8
8-bit KV cache usually costs very little quality and roughly halves cache memory, buying longer context or more concurrency. Best value-per-effort step available.
Technique 3: speculative decoding
A small draft model proposes the next few tokens; the large model verifies them in one parallel pass. Accepted tokens come "free"; on the first mismatch you restart from there.
Real speedups commonly land between roughly 1.5x and 3x depending on draft/main agreement. Highly predictable text - code, structured output, templated answers - benefits most; open-ended creative writing barely at all. llama.cpp also supports model-free lookup decoding via n-gram matching for templated output. Note it does not save VRAM; it adds a second model.
Technique 4: prefix caching
System prompts, tool definitions and few-shot examples repeat in every request. Prefix caching computes their KV once and reuses it. For agent workloads with thousands of tokens of tool schemas, this is pure waste elimination. Most frameworks expose it as a flag; enabling it typically drops TTFT noticeably.
Technique 5: weight quantization
4-bit weights halve memory traffic, so generation approaches double speed in theory. Real gains depend on dequantization overhead and hardware low-bit support - usually solid, rarely linear.
Combinations by scenario
| Scenario | Stack | Expected |
|---|---|---|
| Single local user | Q4 weights + q8 KV + prefix cache | Half the VRAM, longer context |
| Small team API | vLLM/SGLang + continuous batching + prefix cache | Several-fold throughput |
| Templated output (JSON, completion) | Above + speculative/lookup decoding | Another 1-2x |
| Long-document Q&A | Quantized KV + chunking | Longer context becomes feasible |
Flags worth knowing
| Tool | Flag | Effect |
|---|---|---|
| vLLM | --max-num-seqs |
Concurrent sequence cap |
| vLLM | --gpu-memory-utilization |
VRAM fraction; leave headroom |
| vLLM | --enable-prefix-caching |
Enable prefix reuse |
| llama.cpp | -ngl |
GPU layers; if not maxed you are not using the GPU |
| llama.cpp | -b / -ub |
Batch and micro-batch size for prefill |
| llama.cpp | --flash-attn |
Flash Attention for long contexts |
A classic mistake: leave -ngl at default, run most of the model on CPU, then blame the GPU.
Measurement discipline
Change one variable at a time and record TTFT, generation speed and p95 latency over at least dozens of requests. Single measurements are noise. Do not trust absolute numbers in any blog, including the ranges here - they indicate magnitude only.
FAQ
Per-request latency rises with concurrency - normal? Yes. Batching trades latency for throughput. If p95 stays acceptable and total throughput climbs, you won.
Does KV cache quantization hurt quality? 8-bit rarely; 4-bit can break precise recall over long contexts. Start at 8-bit.
Is speculative decoding worth it? Only for predictable output, and only if you can spare the extra VRAM.
GPU utilization is high but it is still slow? Utilization is not efficiency. Look at batch size, memory bandwidth saturation, or CPU-side data preparation. Track time per token instead.
Note: flag names and defaults change between releases; check official documentation. Speedups must be measured on your own hardware and workload.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.