Local LLM Setup on Linux in 2026: Ollama (178k Stars) Install Guide with GPU, Docker and Systemd - Complete Walkthrough

Ollama (177,825 stars) makes local LLMs trivial on Linux: one install script, GPU or CPU, systemd service included. This guide covers install, model pull, GPU checks, Docker and API usage.

💡 What You Will Learn

Ollama (177,825 stars) makes local LLMs trivial on Linux: one install script, GPU or CPU, systemd service included. This guide covers install, model pull, GPU checks, Docker and API usage.

📜 Table of Contents

• The short answer
• Step 1 - Install
• Step 2 - Verify GPU (optional)
• Step 3 - Pull and run a model
• Step 4 - Docker option
• Step 5 - Use the API
• Real numbers
• FAQ

The short answer

On Linux, Ollama (177,825 stars, MIT) is the fastest path to a local LLM: one command installs it, downloads register as systemd services, and the API is OpenAI-compatible so any tool can connect. A 7-8B model runs on CPU (slow) or a consumer GPU (fast).

Step 1 - Install

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama    # should be active (running)

Step 2 - Verify GPU (optional)

ollama ps            # shows loaded models + GPU usage
nvidia-smi           # confirm NVIDIA driver sees your GPU

Ollama auto-detects NVIDIA GPUs via CUDA. AMD users need ROCm setup; CPU-only works out of the box.

Step 3 - Pull and run a model

ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Explain Linux permissions in one paragraph"

Step 4 - Docker option

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Step 5 - Use the API

curl http://localhost:11434/api/generate -d '{"model":"qwen2.5:7b","prompt":"hi"}'

Any OpenAI-compatible client can also point at http://localhost:11434/v1.

Real numbers

• Ollama is the most-starred local LLM tool on GitHub (177,825 stars) with models from 0.5B to 70B+.
• 7-8B Q4 quantized models need ~6GB RAM; 13B needs ~9GB; 70B needs ~40GB.
• A 7B model on an RTX 3060 (12GB): ~40-60 tokens/sec. On CPU-only: ~5-10 tokens/sec.

FAQ

Q: Which model should I start with? A: qwen2.5:7b or llama3.1:8b - the best quality-per-GB balance for 8GB+ machines.

Q: How do I uninstall? A: sudo systemctl stop ollama then remove /usr/local/bin/ollama and ~/.ollama.

Q: Can multiple users share it? A: Yes - it runs as a systemd service listening on localhost; expose with caution (add auth if binding to 0.0.0.0).

❓ FAQ

Which model should I start with?

qwen2.5:7b or llama3.1:8b - the best quality-per-GB balance for 8GB+ machines.

How do I uninstall?

`sudo systemctl stop ollama` then remove `/usr/local/bin/ollama` and `~/.ollama`.

Can multiple users share it?

Yes - it runs as a systemd service listening on localhost; expose with caution (add auth if binding to 0.0.0.0).