Running Ollama in Docker gives a portable LLM server: one command to start, models in a volume, an API on 11434. This guide covers setup, GPU passthrough, persistence, Compose, and the Mac GPU caveat.

💡 What You Will Learn

Running Ollama in Docker gives a portable LLM server: one command to start, models in a volume, an API on 11434. This guide covers setup, GPU passthrough, persistence, Compose, and the Mac GPU caveat.

📜 Table of Contents

• Why containerize it
• Requirements
• Three steps
• GPU
• Mac caveat
• Common flags
• Upgrade and add a UI
• FAQ

Run Ollama Locally with Docker: Complete Setup Guide

Docker plus Ollama is the least painful way to self-host a local LLM: one command starts the server, models live in a volume you can move between machines, and the API sits on port 11434.

Why containerize it

Isolation from host dependencies, version control by image tag, a stable endpoint on 11434 that any script or chat UI can call, and portable data - back up the volume and you have backed up every model. The cost is a small overhead, roughly 1-2% on Linux.

Requirements

Docker Engine or Docker Desktop, an NVIDIA driver plus the NVIDIA Container Toolkit on Linux, and disk space (about 4-5GB for a 7B model, 9GB for 14B).

Three steps

docker run -d --name ollama -p 11434:11434 \
  -v ollama_data:/root/.ollama --restart unless-stopped ollama/ollama
docker exec -it ollama ollama pull llama3.1:8b
curl http://localhost:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"hi"}'

Ollama also exposes an OpenAI-compatible endpoint at /v1/chat/completions.

GPU

Add --gpus all. If nvidia-smi shows nothing inside the container, the host is missing the NVIDIA Container Toolkit. AMD users need a ROCm image; check the docs for current tags.

Mac caveat

Docker Desktop for Mac has no GPU passthrough, so the container falls back to CPU and runs far slower than a native install. On Mac, install natively - the port stays 11434 and no client code changes.

Common flags

Upgrade and add a UI

Rewriting the command as compose.yaml makes upgrades a pull and recreate; GPU reservations go under deploy.resources. For chat, pair it with Open WebUI on port 3000 pointed at host.docker.internal:11434. In your own code set base_url=http://localhost:11434/v1 and any non-empty API key.

FAQ

Can a native Ollama reuse container models? Yes - both read ~/.ollama, so map the volume to the same host path.

Why is the first call slow? The model is loading into VRAM or RAM.

Is exposing 11434 safe? No. Ollama has no auth; bind to localhost or add an authenticating proxy.

Image tags and flags change - check official docs.