Ollama is the easiest way to run local LLMs, but Windows users need specific steps for GPU acceleration, model management, and integrating with tools like Open WebUI and VS Code.

💡 What You Will Learn

Ollama is the easiest way to run local LLMs, but Windows users need specific steps for GPU acceleration, model management, and integrating with tools like Open WebUI and VS Code.

📜 Table of Contents

• Step 1: Install Ollama on Windows
• Step 2: Enable GPU Acceleration (NVIDIA)
• Step 3: Pull and Run Models
• Step 4: Set Up Open WebUI
• Step 5: Connect to VS Code (Continue extension)
• Performance Benchmarks on Windows

Ollama Windows Setup Guide: Install and Run Local LLMs in 10 Minutes

Ollama (176K GitHub stars) makes running local LLMs on Windows dead simple.

Step 1: Install Ollama on Windows

Download the official Windows installer from ollama.com/download. Run it. The installer adds Ollama to your system PATH automatically. After installation, verify:

ollama --version

Step 2: Enable GPU Acceleration (NVIDIA)

Ollama automatically detects NVIDIA GPUs via CUDA. GPU Memory Requirements by Model Size: | Model Size | GPU VRAM Needed | Example GPUs | |-----------|----------------|-------------| | 7B Q4 | ~5GB | RTX 3060 12GB, RTX 4060 | | 13B Q4 | ~8GB | RTX 3070, RTX 4070 | | 34B Q4 | ~20GB | RTX 3090, RTX 4090 | | 70B Q4 | ~40GB | Dual RTX 3090, A6000 |

Step 3: Pull and Run Models

ollama pull llama3.1:8b
ollama run llama3.1:8b

Step 4: Set Up Open WebUI

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 — a ChatGPT clone powered by your local LLM.

Step 5: Connect to VS Code (Continue extension)

Install Continue extension in VS Code. In config.json:

{
  "models": [{
    "title": "Ollama Local",
    "provider": "ollama",
    "model": "llama3.1:8b"
  }]
}

Performance Benchmarks on Windows