Ollama in Docker With a GPU: The Compose Block, the Env Defaults, and the VRAM Math Behind Silent CPU Offload

A container that serves the first prompt and then crawls at four tokens per second is not slow hardware, it is a model split across GPU and CPU with no error raised. The device reservation block, the env defaults read from the source, and the arithmetic that predicts an OOM before it happens.

💡 What You Will Learn

A container that serves the first prompt and then crawls at four tokens per second is not slow hardware, it is a model split across GPU and CPU with no error raised. The device reservation block, the

📜 Table of Contents

• First, prove the GPU is inside the container
• The compose file that keeps the model on the GPU
• Environment defaults, as they exist in the binary
• Context length is decided for you, and the default is small
• The VRAM math, with real file sizes
• Verify the split, not just the answer
• Failure modes, mapped
• What to change first, in order
• FAQ
• Does the official image include CUDA?
• Why does ollama ps show a CPU percentage I never asked for?
• How much VRAM does a 14B model need at 32k context?
• Does a quantized K/V cache help?
• Can I run two models at once?
• Related reading

A container that starts cleanly, answers the first prompt, then crawls at four tokens per second is the most common Ollama-on-Docker failure, and it is not a performance bug. The model was split between GPU and CPU, and nothing on screen said so. The image cannot tell you the GPU was never attached; you have to ask it.

This is the working configuration, the environment defaults that actually ship in the binary (read from envconfig/config.go), and the arithmetic that predicts an out-of-memory error before you hit it.

First, prove the GPU is inside the container

The ollama/ollama image ships CUDA and ROCm libraries, but it does not ship the driver. On an NVIDIA host that means the Container Toolkit, in this order:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo -S -p '' gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo -S -p '' tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo -S -p '' apt-get update
sudo -S -p '' apt-get install -y nvidia-container-toolkit
sudo -S -p '' nvidia-ctk runtime configure --runtime=docker
sudo -S -p '' systemctl restart docker

Then the one-line smoke test. If this prints nothing or errors, stop here, because every model you pull will run on CPU:

docker run --rm --gpus=all ollama/ollama nvidia-smi -L

Two variants worth knowing. On AMD, skip the toolkit and use the rocm image tag with the device nodes passed through: --device /dev/kfd --device /dev/dri. On a Jetson board, Ollama cannot detect the JetPack version, so pass JETSON_JETPACK=5 or 6 explicitly. Vulkan is bundled in the standard image and switches on when the container can see the devices; OLLAMA_VULKAN=0 turns it off, and GGML_VK_VISIBLE_DEVICES picks specific Vulkan devices.

The compose file that keeps the model on the GPU

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      OLLAMA_KEEP_ALIVE: "30m"
      OLLAMA_NUM_PARALLEL: "2"
      OLLAMA_MAX_LOADED_MODELS: "1"
      OLLAMA_CONTEXT_LENGTH: "32768"
      OLLAMA_FLASH_ATTENTION: "1"
      OLLAMA_KV_CACHE_TYPE: "f16"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 30s
      timeout: 5s
      retries: 3

volumes:
  ollama:

The deploy.resources.reservations.devices block is the compose v2 equivalent of --gpus=all. Drop it and the container starts, serves requests, and quietly runs on CPU. That single omission accounts for most "Ollama is slow in Docker" reports.

Environment defaults, as they exist in the binary

The KEEP_ALIVE default of five minutes is why a chat UI feels fast and then slow: the first request after an idle gap pays the full load cost. In a container that restarts with the host, set 30m or a negative value and the reload disappears.

Context length is decided for you, and the default is small

Ollama picks a context length from available VRAM at load time:

A 16 GB card therefore gives the model four thousand tokens of memory. A coding assistant, an agent loop or a long-document prompt quietly gets truncated, and the symptom looks like "the model forgot the earlier instructions". The documentation's own guidance is that agent, coding and web-search workloads want at least 64,000 tokens, which means an explicit OLLAMA_CONTEXT_LENGTH=64000 plus the VRAM to hold the cache.

There is a second cap you cannot override. If you request more context than the model was trained for, the server logs requested context size too large for model and clamps the value to the training length. Raising the variable past that ceiling buys nothing.

The VRAM math, with real file sizes

Two numbers decide whether a model fits: the weight file and the K/V cache. The weight file is fixed and published in the registry. These are the default tags, measured today:

The K/V cache is the part people forget, and it grows linearly with context. Per token, in bytes:

2 (K and V) x layers x KV heads x head_dim x bytes_per_element

For Qwen3-8B, whose config reports 36 layers, 8 KV heads and head_dim 128, at the default f16 precision:

2 x 36 x 8 x 128 x 2 = 147,456 bytes per token = 144 KiB

4,096 tokens  ->   576 MiB
32,768 tokens ->   4.5 GiB
65,536 tokens ->   9.0 GiB

Llama-3.1-8B lands at 32 layers and head_dim 128, so 128 KiB per token and 4 GiB at 32k. Parameter count is not the driver here: both models are 8B, and the cache differs by 12 percent because the layer and head geometry differs. Gemma3-12B reports 48 layers with head_dim 256, which computes to 384 KiB per token, though its sliding-window attention layers mean the real allocation is lower than that formula suggests. Assume the worst case and you will not be surprised.

Then multiply by OLLAMA_NUM_PARALLEL, because each slot gets its own window. Two parallel requests at 32k on Qwen3-8B is 9 GiB of cache on top of 5.23 GB of weights, which is exactly how a 16 GB card ends up at 40 percent CPU after you "just" raised concurrency.

Working budgets, leaving a gigabyte for the driver and CUDA context:

Verify the split, not just the answer

Every diagnostic you need is one command, and the PROCESSOR column is the one that matters:

docker exec ollama ollama ps

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3:8b         bdbd181c33f2    9.7 GB    100% GPU     32768      29 minutes from now

100% GPU means the whole model and its cache are resident. A split line such as 62%/38% CPU is your out-of-memory warning, already applied: the loader kept what fit and spilled the rest, with no error anywhere. Alongside it, ollama ps reports the context actually allocated, which is the fastest way to catch a truncated default.

For a JSON view from outside the container, curl -s localhost:11434/api/ps returns the same fields. Wire that into the healthcheck if you want the container marked unhealthy for CPU-only serving.

Failure modes, mapped

What to change first, in order

If you are setting this up today, the sequence that avoids every trap above is short: prove the GPU is visible inside the container, pin the context length explicitly instead of trusting the VRAM heuristic, set KEEP_ALIVE longer than your traffic gap, and keep NUM_PARALLEL at 1 until you have measured the cache cost. Everything else is tuning. Those four decide whether the deployment is usable at all.

FAQ

Does the official image include CUDA?

Yes, the ollama/ollama image bundles the CUDA and ROCm runtime libraries. What it cannot bundle is the host driver and the container runtime hook, which is why the Container Toolkit step is mandatory on NVIDIA hosts.

Why does ollama ps show a CPU percentage I never asked for?

Because the model and its K/V cache did not both fit. The loader offloads the remainder to system RAM and continues, trading latency for the ability to run at all. There is no error, so the PROCESSOR column is the only signal.

How much VRAM does a 14B model need at 32k context?

Roughly 13.8 GB with the default qwen3:14b tag: 9.28 GB of weights plus about 4.5 GiB of f16 K/V cache. That fits a 16 GB card only at concurrency 1, with nothing left over for a second model.

Does a quantized K/V cache help?

OLLAMA_KV_CACHE_TYPE defaults to f16. A narrower cache cuts the numbers in the formula above, at some cost in output quality. Measure the quality loss on your own prompts before shipping it; it is the wrong first lever for a memory problem that a smaller model would solve.

Can I run two models at once?

Only if both fit. Weights plus cache for each must coexist in VRAM, and two 8B models at 32k is over 19 GB before overhead. On a single card, one model with a longer keep-alive is almost always the better trade.

Related reading

• Running Ollama locally with Docker: the basics
• Ollama complete guide: models, Modelfiles and API
• vLLM deployment guide: when to leave Ollama
• Ollama vs vLLM vs TGI: throughput and memory compared