AI Inference Infrastructure 2026: GPUs, Memory and the Stack That Makes Models Fast
Why is one team serving the same model 10x faster than another? Usually not the GPU - it is the infrastructure around it. Here is the inference stack, from silicon to scheduler, and where the real bottlenecks live.
💡 What You Will Learn
Why is one team serving the same model 10x faster than another? Usually not the GPU - it is the infrastructure around it. Here is the inference stack, from silicon to scheduler, and where the real bot
📜 Table of Contents
The Stack, Top to Bottom
Inference infrastructure is five layers, and most optimization wins come from the top three, not from buying new hardware (stars fetched 2026-08-12).
Layer 1: The model format. Quantized weights (GGUF, AWQ, GPTQ) shrink memory 2-4x. llama.cpp (123,466 stars) and its GGUF format made 70B-class models runnable on a single workstation.
Layer 2: The inference engine. vLLM (88,784 stars) with PagedAttention and continuous batching is the default; SGLang (31,684 stars) wins on some structured workloads. Both are 5-20x faster than naive Transformers generate() loops.
Layer 3: The scheduler. Continuous batching - packing requests onto the GPU as slots free - is the single biggest throughput lever. This is engine-level, which is why you should not hand-roll serving.
Layer 4: The hardware. VRAM size decides what fits; memory bandwidth decides token speed. For 7B-13B models, a 24 GB consumer GPU is the price/performance sweet spot. For 70B+, you need 2x 48 GB pro cards or quantization plus patience.
Layer 5: The network. For single-node serving, irrelevant. For multi-node, NVLink/InfiniBand bandwidth becomes the constraint.
The Real Bottleneck Order
- Memory bandwidth - tokens/second is bounded by how fast weights stream from VRAM, not by FLOPs. This is why quantization helps speed, not just size.
- VRAM capacity - decides which model fits at all.
- Batch efficiency - decides how many users share one GPU.
- Latency tail - p99 matters for chat; the scheduler and queue depth control it.
A Sane Reference Setup (2026)
- 7B model, one RTX 4090 (24 GB), vLLM, GGUF or AWQ 4-bit: 50-200 concurrent users, ~1,500-3,000 tok/s aggregated.
- 70B model, two 48 GB pro GPUs, vLLM with tensor parallelism: dozens of users, ~500-1,000 tok/s.
- Budget option: rent, don't buy - 4090s rent for ~$0.30-0.50/hour in 2026.
The Mistakes That Cost 10x
- Serving with naive Transformers code instead of an engine (5-20x loss).
- No batching (throughput collapses under load).
- Oversized context windows (attention cost grows superlinearly with input).
- Buying GPUs before quantizing (4-bit often halves the fleet).
FAQ
What is the most important spec for inference? Memory bandwidth first, then VRAM capacity. FLOPs matter least for token generation.
CPU or GPU for inference? GPU for anything concurrent; CPU (llama.cpp) only for single-user or offline workloads.
How many users can one GPU serve? A 24 GB card with a 7B 4-bit model and vLLM typically handles 50-200 concurrent users depending on context length.
Related reads: AI Inference 2026, AI Model Serving 2026, Local LLM Hardware Guide 2026.
❓ FAQ
What is the most important spec for inference?
Memory bandwidth first, then VRAM capacity. FLOPs matter least for token generation.
CPU or GPU for inference?
GPU for anything concurrent; CPU (llama.cpp) only for single-user or offline workloads.
How many users can one GPU serve?
24 GB card with a 7B 4-bit model and vLLM typically handles 50-200 concurrent users depending on context length.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
