Local Large Language Model Linux Setup: Run 7B to 70B Models on Ubuntu
🩺 Summary
Linux is the best OS for running local LLMs: better GPU support, lower overhead, more tools. This guide covers installing Ollama, llama.cpp, CUDA, and Docker on Ubuntu.
📝 Details
# Local Large Language Model Linux Setup: Run 7B to 70B Models on Ubuntu
Linux is the preferred OS for local LLM inference. Better GPU drivers, lower memory overhead, and full toolchain access.
## Step 1: Install NVIDIA Drivers and CUDA
```bash
# Check GPU
lspci | grep -i nvidia
# Add NVIDIA driver repository
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-driver-550
sudo reboot
# Verify
nvidia-smi
```
## Step 2: Install Ollama (Easiest)
```bash
curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama
ollama pull llama3.1:8b
ollama run llama3.1:8b "Hello!"
```
Ollama runs as a systemd service automatically.
## Step 3: Install llama.cpp (Power Users)
```bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_CUDA=ON
cmake --build build --config Release -j
# Download GGUF model
pip install huggingface-hub
huggingface-cli download TheBloke/Qwen2.5-7B-Instruct-GGUF \
qwen2.5-7b-instruct.Q4_K_M.gguf --local-dir ./models
# Run
./build/bin/llama-cli -m ./models/qwen2.5-7b-instruct.Q4_K_M.gguf \
-p "Hello" -n 128 -ngl 99
```
## Step 4: Docker Setup
```bash
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
# NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Run Ollama in Docker with GPU
docker run -d --gpus all -v ollama_data:/root/.ollama \
-p 11434:11434 --name ollama ollama/ollama
```
## Step 5: Complete AI Stack
```bash
# Install everything with one script
sudo apt install -y python3-pip python3-venv build-essential cmake git
curl -fsSL https://ollama.com/install.sh | sh
pip install torch transformers langchain chromadb sentence-transformers
# Pull models
ollama pull llama3.1:8b
ollama pull nomic-embed-text
```
## Performance: Linux vs Windows vs Mac
| OS | 7B Q4 Speed | GPU Support | RAM Overhead |
|---|-----------|-------------|-------------|
| Linux (Ubuntu) | 100% (baseline) | CUDA/ROCm/Metal | ~1GB |
| Windows | 95-98% | CUDA/DirectML | ~3GB |
| macOS | 90-95% | Metal (MPS) | ~2GB |
## System Optimization
```bash
echo 'vm.max_map_count=2147483642' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
sudo swapoff -a # Prevent memory thrashing
```
## FAQ
**Q: Ubuntu 22.04 or 24.04?** A: 24.04 LTS for better GPU driver support.
**Q: Run Ollama on startup?** A: It's automatic via systemd.
**Q: CUDA without a display?** A: Yes, NVIDIA drivers work headless.
💬 Comments (0)