LLM Context Window 2026: 128K vs 1M Tokens and What the Real Limits Are
Models advertise 128K, 200K, even 1M token contexts, but quality degrades well before the advertised limit. Here is how to measure your real usable window, how to size the KV cache, and what to do beyond it.
📜 Table of Contents
LLM Context Window 2026: 128K vs 1M Tokens and the Real Limits
Models advertise 128K, 200K, even 1M token contexts, but quality degrades well before the advertised limit.
Two Different Numbers
- Advertised window: the maximum the model accepts before erroring.
- Usable window: where quality still holds, usually 30-60% of the advertised size.
| Advertised | Usable (typical) | Symptom |
|---|---|---|
| 32K | 16-24K | late-document details drop |
| 128K | 32-64K | middle facts ignored |
| 200K | 64-100K | cross-section reasoning breaks |
| 1M | 100-300K | varies; retrieval tasks more robust than reasoning |
Quality Limit: the Middle Is a Dead Zone
Models use the start and end of long contexts far better than the middle - the lost-in-the-middle effect. Practical consequences: put instructions, output formats and key evidence at the edges; structure long material with clear delimiters; and hand needle-in-haystack lookups to retrieval instead of the model.
Memory Limit: KV Cache Arithmetic
KV bytes ≈ 2 (K and V) × layers × KV heads × head dim × seq len × bytes per element
Worked example - 32 layers, 8 KV heads, head dim 128, fp16:
| Context | KV cache |
|---|---|
| 8K | ~1 GiB |
| 32K | ~4 GiB |
| 128K | ~16 GiB |
| 1M | ~122 GiB |
So "1M context" on a consumer card usually means very slow or OOM. Full multi-head attention with 32 KV heads quadruples this again.
Measure Your Own Usable Window
- Build long inputs at 8K/16K/32K/64K/128K.
- Insert a unique fact at 10/25/50/75/90% positions.
- Ask the same question each time.
- Record accuracy as a matrix.
- The tier where accuracy drops below ~90% is your usable window.
Dozens of calls, and it beats any marketing page because it runs on your data and hardware.
Beyond the Window
| Technique | How | Cost |
|---|---|---|
| RAG | retrieve 2-4K relevant chunks | needs good chunking |
| Rolling summary | summarise old turns, keep recent verbatim | detail loss |
| Hierarchical | summarise per section, then summarise summaries | extra calls |
| KV quantisation | lower cache precision | framework support, quality risk |
| Prefix caching | reuse a fixed long prefix | only for repeated prefixes |
FAQ
Do 1M windows remove the need for RAG? No - cost, latency, and the middle still gets ignored.
Why do answers differ on the same long input? Sampling plus position sensitivity. Lower temperature, fix ordering, front-load key facts.
Can prompting expand the usable window? Slightly. It does not change the order of magnitude.
Sizing local VRAM? Weights + KV cache + overhead. Compute KV first, add quantised weights and 10-20% headroom.
Note: actual windows and cache implementations follow model cards and inference framework docs.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.