llama.cpp runs Llama-class models on laptops, Raspberry Pis and phones: GGUF quantization, CPU/GPU hybrid inference, and a built-in OpenAI-compatible server. This guide goes from build to a working endpoint, with every parameter explained.

💡 What You Will Learn

llama.cpp runs Llama-class models on laptops, Raspberry Pis and phones: GGUF quantization, CPU/GPU hybrid inference, and a built-in OpenAI-compatible server. This guide goes from build to a working en

📜 Table of Contents

• Three things to settle first
• Build
• Pick a quantization
• First run
• Serve it
• Memory estimate
• FAQ

Run Llama Locally in 2026: Complete llama.cpp Guide

llama.cpp is the reference implementation of the GGUF format. It is plain C/C++ with no Python dependency, and a 4-bit quantized model cuts memory to roughly a quarter of FP16, which is how a single machine runs weights it could never hold at full precision.

Three things to settle first

1. GGUF is a file format, not a model. The same Llama weights ship as several quantization levels; quality trades against size.
2. llama.cpp does not download anything. Models come from community repos on Hugging Face.
3. CPU-only works. GPU offload is an accelerator, not a requirement.

Build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON      # or METAL / VULKAN / plain CPU
cmake --build build --config Release -j

Binaries land in build/bin/: llama-cli for chat, llama-server for HTTP. Windows users can grab the prebuilt Release and skip the toolchain.

Pick a quantization

Fit by file size first, then quality. Below 3 bits you get repetition and ignored instructions; that is quantization damage, not a prompt problem.

First run

./build/bin/llama-cli -m ./models/model-Q4_K_M.gguf -ngl 99 -c 4096 -t 8 -p "Explain quantization in three sentences"

-ngl 99 offloads all layers to GPU; drop the flag for pure CPU. -c sets context, -t sets CPU threads, -fa enables Flash Attention.

Serve it

./build/bin/llama-server -m ./models/model-Q4_K_M.gguf -ngl 99 -c 8192 --host 0.0.0.0 --port 8080

Then call it with the OpenAI chat-completions shape at /v1/chat/completions; the server also ships a web UI and a native /completion route.

Memory estimate

Weights ≈ parameters × bits ÷ 8. An 8B model at Q4_K_M is roughly a 5GB file. On top of that sits the KV cache, which grows linearly with context: doubling context roughly doubles that share. When VRAM is tight, cut context before you cut quantization.

FAQ

CPU only? Yes, a few tokens per second for 7B on a modern desktop; Apple Silicon's unified memory does much better.

Endless repetition? Raise repeat penalty toward 1.15, lower temperature, and check for loops in the prompt format.

Multiple models? Run several llama-server instances on different ports; each holds its own memory.

Versus Ollama? Ollama wraps the same inference library and adds model management. Use Ollama for convenience, llama.cpp for control.

Parameter names and defaults move between releases; check the repo README. File sizes vary by repo.

❓ FAQ

What is GGUF?

The file format that stores quantized model weights + metadata, designed by the llama.cpp project.

Which model size for my RAM?

Rule of thumb: pick a quantized model roughly half your free RAM. 8GB RAM - 4B models; 16GB - 8B; 32GB - 13-14B.

Can I use a GPU?

Yes - CUDA, ROCm, Metal and Vulkan backends are supported. Use `-ngl 99` to offload all layers to GPU.