Linux is the best OS for running local LLMs: better GPU support, lower overhead, more tools. This guide covers installing Ollama, llama.cpp, CUDA, and Docker on Ubuntu.

💡 What You Will Learn

Linux is the best OS for running local LLMs: better GPU support, lower overhead, more tools. This guide covers installing Ollama, llama.cpp, CUDA, and Docker on Ubuntu.

📜 Table of Contents

• Step 1: Install NVIDIA Drivers and CUDA
• Step 2: Install Ollama (Easiest)
• Step 3: Install llama.cpp (Power Users)
• Step 4: Docker Setup
• Step 5: Complete AI Stack
• Performance: Linux vs Windows vs Mac
• FAQ

Local Large Language Model Linux Setup: Run 7B to 70B Models on Ubuntu

Linux is the preferred OS for local LLM inference. Better GPU drivers, lower memory overhead, and full toolchain access.

Step 1: Install NVIDIA Drivers and CUDA

# Check GPU
lspci | grep -i nvidia

# Add NVIDIA driver repository
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-driver-550
sudo reboot

# Verify
nvidia-smi

Step 2: Install Ollama (Easiest)

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama
ollama pull llama3.1:8b
ollama run llama3.1:8b "Hello!"

Ollama runs as a systemd service automatically.

Step 3: Install llama.cpp (Power Users)

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_CUDA=ON
cmake --build build --config Release -j

# Download GGUF model
pip install huggingface-hub
huggingface-cli download TheBloke/Qwen2.5-7B-Instruct-GGUF \
    qwen2.5-7b-instruct.Q4_K_M.gguf --local-dir ./models

# Run
./build/bin/llama-cli -m ./models/qwen2.5-7b-instruct.Q4_K_M.gguf \
    -p "Hello" -n 128 -ngl 99

Step 4: Docker Setup

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

# NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Run Ollama in Docker with GPU
docker run -d --gpus all -v ollama_data:/root/.ollama \
    -p 11434:11434 --name ollama ollama/ollama

Step 5: Complete AI Stack

# Install everything with one script
sudo apt install -y python3-pip python3-venv build-essential cmake git
curl -fsSL https://ollama.com/install.sh | sh
pip install torch transformers langchain chromadb sentence-transformers

# Pull models
ollama pull llama3.1:8b
ollama pull nomic-embed-text

Performance: Linux vs Windows vs Mac

echo 'vm.max_map_count=2147483642' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
sudo swapoff -a  # Prevent memory thrashing

FAQ

Q: Ubuntu 22.04 or 24.04? A: 24.04 LTS for better GPU driver support. Q: Run Ollama on startup? A: It's automatic via systemd. Q: CUDA without a display? A: Yes, NVIDIA drivers work headless.

❓ FAQ

Ubuntu 22.04 or 24.04?

24.04 LTS for better GPU driver support.

CUDA without a display?

Yes, NVIDIA drivers work headless.