Run Llama Locally in 2026: Complete llama.cpp (123k Stars) Guide - From GGUF Download to Chat on Any Hardware

llama.cpp (122,773 stars, MIT) runs Llama-class models on almost anything - laptop, Raspberry Pi, phone. This guide covers GGUF formats, quantization levels, CPU/GPU builds and the chat CLI.

💡 What You Will Learn

llama.cpp (122,773 stars, MIT) runs Llama-class models on almost anything - laptop, Raspberry Pi, phone. This guide covers GGUF formats, quantization levels, CPU/GPU builds and the chat CLI.

📜 Table of Contents

• The short answer
• Step 1 - Get llama.cpp
• Step 2 - Download a GGUF model
• Step 3 - Chat
• Understanding quantization
• Real numbers
• FAQ

The short answer

llama.cpp (122,773 stars, MIT) is the reference C/C++ runtime for Llama-class models. Its superpower is quantization: models converted to GGUF format at 4-bit precision run on machines that could never fit the original FP16 weights.

Step 1 - Get llama.cpp

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # add for NVIDIA GPU
cmake --build build --config Release -j

CPU-only: skip the CUDA flag. macOS: it works with Metal out of the box.

Step 2 - Download a GGUF model

# Example: Llama 3.1 8B, 4-bit quantized (from Hugging Face)
wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

Step 3 - Chat

./build/bin/llama-cli -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p "Explain quantum computing simply" -n 256

Understanding quantization

Real numbers

• Q4_K_M is the community default: ~4.9GB for 8B models, runs on 8GB RAM machines.
• On an M1/M2 Mac: 8B Q4 runs at ~20-30 tokens/sec. On a 4090: 100+ tokens/sec.
• Raspberry Pi 5 can run 3-4B models at usable speeds (~5-10 tokens/sec).

FAQ

Q: What is GGUF? A: The file format that stores quantized model weights + metadata, designed by the llama.cpp project.

Q: Which model size for my RAM? A: Rule of thumb: pick a quantized model roughly half your free RAM. 8GB RAM - 4B models; 16GB - 8B; 32GB - 13-14B.

Q: Can I use a GPU? A: Yes - CUDA, ROCm, Metal and Vulkan backends are supported. Use -ngl 99 to offload all layers to GPU.

❓ FAQ

What is GGUF?

The file format that stores quantized model weights + metadata, designed by the llama.cpp project.

Which model size for my RAM?

Rule of thumb: pick a quantized model roughly half your free RAM. 8GB RAM - 4B models; 16GB - 8B; 32GB - 13-14B.

Can I use a GPU?

Yes - CUDA, ROCm, Metal and Vulkan backends are supported. Use `-ngl 99` to offload all layers to GPU.