Run LLM on CPU 2026: Models, Speeds and Setup for Machines Without a GPU
No GPU, no budget for cloud, but you still want local LLMs. What actually runs on CPU, how fast, and what is the honest tradeoff?
💡 What You Will Learn
No GPU, no budget for cloud, but you still want local LLMs. What actually runs on CPU, how fast, and what is the honest tradeoff?
📜 Table of Contents
CPU Inference Is Real, With Limits
llama.cpp (123,325 stars) and Ollama (178,206 stars) both run GGUF models on CPU. The physics: a modern CPU delivers roughly 5-20 tokens per second for a 7-8B Q4 model. That is slow but usable for chat, summarization, and batch jobs. It is not usable for real-time assistants.
What Runs Well on CPU (2026)
| Model size | Quant | RAM needed | Speed (8-core) | Use for |
|---|---|---|---|---|
| 1-3B | Q4 | 2-4GB | 30-60 tok/s | classification, extraction, fast tasks |
| 7-8B | Q4 | 5-7GB | 8-15 tok/s | chat, drafting, summarization |
| 13-14B | Q4 | 9-11GB | 4-7 tok/s | higher quality chat |
| 32B+ | Q4 | 20GB+ | 1-3 tok/s | only for patient batch jobs |
Setup: Two Commands With Ollama
# install then pull a small model
ollama pull qwen3:1.5b # about 1GB, fast
ollama run qwen3:1.5b
For more control, llama.cpp directly:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j
./build/bin/llama-cli -m model.gguf -p "Hello" -n 50
The Three Levers
- Model size - biggest lever. 7B Q4 vs 1.5B Q4 is a 4-6x speed difference.
- Quantization - Q4_K_M is the sweet spot; Q8 is about 30% slower for small quality gain, Q2 is fast but visibly worse.
- Threads & hardware - llama.cpp uses AVX2/AVX512. Set -t to your core count. Apple Silicon with Metal acceleration runs 2-3x faster than x86.
When CPU Is the Right Call
- Privacy requirements (medical, legal text stays local)
- Learning/experiments on a laptop
- Batch processing where 10 tok/s is fine
- Cost: a 7B Q4 model runs forever on hardware you already own
Honest Bottom Line
If you need conversational-speed responses, rent a GPU for $0.10-0.30/hour instead. If you need private, cheap, offline inference at 10 tok/s, CPU is a completely valid 2026 answer.
