LLM Inference Optimization 2026: Batching, KV Cache and Speculative Decoding Explained
Your self-hosted model serves 2 tokens per second and the GPU is half idle. The big wins in inference speed come from a few well-understood techniques. What are they and how much do they actually help?
💡 What You Will Learn
Your self-hosted model serves 2 tokens per second and the GPU is half idle. The big wins in inference speed come from a few well-understood techniques. What are they and how much do they actually help
📜 Table of Contents
The Bottleneck Nobody Sees
LLM inference is memory-bound, not compute-bound: generating each token streams the whole model through memory. This one fact explains every optimization below - they all reduce memory traffic or reuse what is already loaded.
Technique 1: Continuous Batching
Instead of processing one request at a time, the server interleaves many requests in the same forward pass. This is the single biggest throughput win - engines like vLLM (88,691 stars) and SGLang (31,628 stars) report 10-24x throughput versus naive serving. Latency per request barely changes; total requests per second jumps.
Technique 2: KV Cache
Every generated token attends to all previous tokens; the engine caches those attention keys/values instead of recomputing them. Without a KV cache, cost per token grows linearly with sequence length. Modern engines also quantize the KV cache (4-bit or 8-bit) to fit longer contexts in memory - llama.cpp and vLLM both support it.
Technique 3: Speculative Decoding
A small draft model proposes the next several tokens; the big model verifies them in parallel. When the draft is right (60-80% of the time with a well-matched draft), you generate several tokens in the time of one. Practical speedups: 1.5-3x on consumer hardware, higher on some workloads.
Technique 4: Quantization and Weight Sharing
The quantization guide covers the quality side; the speed side is that 4-bit weights halve memory traffic. Some models also use grouped-query attention (GQA), which shrinks the KV cache - a big deal for long contexts.
How Much Does Each Help (Realistic)
| Technique | Throughput gain | Effort |
|---|---|---|
| Continuous batching | 10-24x | switch to vLLM/SGLang |
| KV cache (already in engines) | baseline | none |
| KV cache quantization | 1.2-2x context capacity | config flag |
| Speculative decoding | 1.5-3x | add draft model |
| 4-bit quantization | ~2x memory traffic | convert model |
The Practical Playbook
- If you serve an API: switch to vLLM or SGLang today - batching alone changes everything.
- If you run local single-user: llama.cpp with a Q4 model plus speculative decoding is the max you will get without a bigger GPU.
- Measure tokens/sec and p95 latency before and after each change; do not trust blog numbers on your hardware.
