Local LLM Setup on Linux in 2026: Ollama (178k Stars) Install Guide with GPU, Docker and Systemd - Complete Walkthrough

Ollama (177,825 stars) makes local LLMs trivial on Linux: one install script, GPU or CPU, systemd service included. This guide covers install, model pull, GPU checks, Docker and API usage.

💡 What You Will Learn

Ollama (177,825 stars) makes local LLMs trivial on Linux: one install script, GPU or CPU, systemd service included. This guide covers install, model pull, GPU checks, Docker and API usage.

## The short answer On Linux, Ollama (177,825 stars, MIT) is the fastest path to a local LLM: one command installs it, downloads register as systemd services, and the API is OpenAI-compatible so any tool can connect. A 7-8B model runs on CPU (slow) or a consumer GPU (fast). ## Step 1 - Install ```bash curl -fsSL https://ollama.com/install.sh | sh systemctl status ollama # should be active (running) ``` ## Step 2 - Verify GPU (optional) ```bash ollama ps # shows loaded models + GPU usage nvidia-smi # confirm NVIDIA driver sees your GPU ``` Ollama auto-detects NVIDIA GPUs via CUDA. AMD users need ROCm setup; CPU-only works out of the box. ## Step 3 - Pull and run a model ```bash ollama pull qwen2.5:7b ollama run qwen2.5:7b "Explain Linux permissions in one paragraph" ``` ## Step 4 - Docker option ```bash docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama ``` ## Step 5 - Use the API ```bash curl http://localhost:11434/api/generate -d '{"model":"qwen2.5:7b","prompt":"hi"}' ``` Any OpenAI-compatible client can also point at `http://localhost:11434/v1`. ## Real numbers - Ollama is the most-starred local LLM tool on GitHub (177,825 stars) with models from 0.5B to 70B+. - 7-8B Q4 quantized models need ~6GB RAM; 13B needs ~9GB; 70B needs ~40GB. - A 7B model on an RTX 3060 (12GB): ~40-60 tokens/sec. On CPU-only: ~5-10 tokens/sec. ## FAQ **Q: Which model should I start with?** A: qwen2.5:7b or llama3.1:8b - the best quality-per-GB balance for 8GB+ machines. **Q: How do I uninstall?** A: `sudo systemctl stop ollama` then remove `/usr/local/bin/ollama` and `~/.ollama`. **Q: Can multiple users share it?** A: Yes - it runs as a systemd service listening on localhost; expose with caution (add auth if binding to 0.0.0.0).