Run Llama Locally in 2026: Complete llama.cpp (123k Stars) Guide - From GGUF Download to Chat on Any Hardware

llama.cpp (122,773 stars, MIT) runs Llama-class models on almost anything - laptop, Raspberry Pi, phone. This guide covers GGUF formats, quantization levels, CPU/GPU builds and the chat CLI.

💡 What You Will Learn

llama.cpp (122,773 stars, MIT) runs Llama-class models on almost anything - laptop, Raspberry Pi, phone. This guide covers GGUF formats, quantization levels, CPU/GPU builds and the chat CLI.

## The short answer **llama.cpp** (122,773 stars, MIT) is the reference C/C++ runtime for Llama-class models. Its superpower is quantization: models converted to GGUF format at 4-bit precision run on machines that could never fit the original FP16 weights. ## Step 1 - Get llama.cpp ```bash git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build -DGGML_CUDA=ON # add for NVIDIA GPU cmake --build build --config Release -j ``` CPU-only: skip the CUDA flag. macOS: it works with Metal out of the box. ## Step 2 - Download a GGUF model ```bash # Example: Llama 3.1 8B, 4-bit quantized (from Hugging Face) wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf ``` ## Step 3 - Chat ```bash ./build/bin/llama-cli -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p "Explain quantum computing simply" -n 256 ``` ## Understanding quantization | Quant | Bits/weight | Size (8B) | Quality | |:------|:-----------:|:---------:|:--------| | Q8_0 | 8 | ~8.5GB | Near lossless | | Q4_K_M | 4.5 | ~4.9GB | Recommended | | Q3_K_M | 3.5 | ~4.0GB | Noticeable drop | | Q2_K | 2.5 | ~3.1GB | Only for tests | ## Real numbers - Q4_K_M is the community default: ~4.9GB for 8B models, runs on 8GB RAM machines. - On an M1/M2 Mac: 8B Q4 runs at ~20-30 tokens/sec. On a 4090: 100+ tokens/sec. - Raspberry Pi 5 can run 3-4B models at usable speeds (~5-10 tokens/sec). ## FAQ **Q: What is GGUF?** A: The file format that stores quantized model weights + metadata, designed by the llama.cpp project. **Q: Which model size for my RAM?** A: Rule of thumb: pick a quantized model roughly half your free RAM. 8GB RAM - 4B models; 16GB - 8B; 32GB - 13-14B. **Q: Can I use a GPU?** A: Yes - CUDA, ROCm, Metal and Vulkan backends are supported. Use `-ngl 99` to offload all layers to GPU.