No GPU and no appetite for cloud bills, but you still want local models. What runs on CPU, how fast, and when is it actually worth it?

📜 Table of Contents

• Why CPU works, and why it is slow
• What runs, and how fast
• Three levers, in order of payoff
• Start with Ollama
• Fine control with llama.cpp
• Measure your own speed
• When CPU makes sense
• Five ways to make CPU tolerable
• FAQ
• Related

Run LLM on CPU 2026: Models, Speeds and Setup for Machines Without a GPU

CPU inference is genuinely viable, within limits. llama.cpp and Ollama both run GGUF models without a GPU; whether it is usable depends on how fast you need it.

On the speed numbers: tokens-per-second figures in this article are typical community reports for the listed hardware, gathered from public benchmarks and user posts — not a controlled measurement on a single machine. Real throughput varies with quantisation, context length, batch size and whatever else is using your GPU. Measure your own with ollama run <model> --verbose and read the eval rate at the end.

Why CPU works, and why it is slow

Every generated token streams the whole weight set from memory. GPUs do this with thousands of lanes and high-bandwidth VRAM; a CPU has dozens of cores and roughly an order of magnitude less bandwidth. Speed is basically memory bandwidth ÷ weight size, which is why quantization pays off so directly on CPU: halve the weights, roughly double tokens per second.

Prefill (processing your prompt) is compute-bound and hurts CPU far more, so "read a long document then answer" feels much worse than short question, short answer.

What runs, and how fast

Ranges below depend on CPU generation, memory channels and thread count - treat them as orders of magnitude:

People read roughly 5-8 characters per second. At ~10 tok/s a 7B Q4 model outpaces reading speed and daily chat feels fine. At 5 tok/s you notice the wait. At 32B it becomes fire-and-forget.

Three levers, in order of payoff

Model size dominates: 7B is four to six times slower than 1.5B, and many tasks are solved by 1.5B. Start small, upgrade only if quality fails.

Quantization: use Q4_K_M on CPU. Halving weights nearly doubles speed and the quality cost is acceptable for most work. Drop to Q3 only under memory pressure; avoid Q2.

Threads and instruction sets: llama.cpp uses AVX2/AVX512 where available and builds with those enabled. Set threads to physical core count - hyperthreads usually add scheduling overhead.

Start with Ollama

ollama pull qwen3:1.5b
ollama run qwen3:1.5b

Without a GPU, Ollama runs on CPU automatically. Add --verbose to see tokens per second.

Fine control with llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j
./build/bin/llama-cli -m qwen3-8b-Q4_K_M.gguf -t 8 -c 4096 -n 256 -p "Write a short product blurb"

--mlock matters: if the OS pages the model out, speed collapses to unusable. Locking prevents that at the cost of reserving the memory.

Measure your own speed

llama.cpp prints eval time and prompt eval time; divide by token count. Keep the model file, prompt length and thread count fixed or the comparison is meaningless. Test both a short prompt (generation speed) and a ~2000-token prompt (prefill speed) - the latter is usually the real bottleneck.

When CPU makes sense

Yes: strict privacy, overnight batch jobs over hundreds of documents, learning how inference works, occasional use that does not justify a GPU.

No: real-time multi-turn chat, long-document Q&A, multiple concurrent users, anything at 32B and above.

Five ways to make CPU tolerable

1. Keep prompts short - prefill is the expensive part.
2. Cap output length with -n.
3. Use dual-channel memory; bandwidth is the whole game.
4. Close heavy background apps - there is no headroom.
5. Schedule batch work overnight.

FAQ

Does integrated graphics help? Some iGPUs are usable through llama.cpp's acceleration backends; results vary. Try raising -ngl and measure.

Is loud fan noise normal? Yes. CPU inference is sustained full load; laptops heat up and may throttle. Cap threads or improve cooling.

Enough RAM but still slow? Check memory channels, paging to disk, and thread count. Those three explain most cases.

Can a CPU run 70B? It loads, at under one token per second. Only viable for overnight offline jobs.

Note: speeds are empirical ranges and vary with CPU, memory configuration and software version. Measure on your own machine.

Related

• Running Ollama on Android: four apps compared
• Hardware requirements for local Ollama