Local LLM Setup Guide 2026: Run AI Models on Windows, Mac, or Linux

You want to run AI models locally for privacy and cost savings, but the setup seems complicated.

💡 What You Will Learn

You want to run AI models locally for privacy and cost savings, but the setup seems complicated.

📜 Table of Contents

• VS Code + Continue.dev
• Summary

|-----|------|-----------|------| || 16GB+RAM, RTX 3060+ | Qwen 2.5 (14B), DeepSeek-Coder-V2 || || 32GB+RAM, RTX 4090+ | Llama 3 (70B), Qwen 2.5 (72B) || |-----|------|:----:|:----:| | MacBook Air M1 (8GB) | Qwen2.5-Coder:7B | 12 tok/s || || Llama 3.1:8B | 8 tok/s || || Qwen 2.5:14B | 15 tok/s || || Llama 3:70B (Q4) | 5 tok/s || Windows Mac

brew install ollama

Linux

curl -fsSL https://ollama.com/install.sh | sh

# 
ollama pull llama3.1:8b

# 
ollama pull qwen2.5:7b

# Conversation
ollama run llama3.1:8b

|-----|---------|:----:|:-------:| || Qwen 2.5 | 7B-14B | 4-8GB | || Llama 3.1 | 8B | 4GB | || DeepSeek-Coder-V2 | 16B | 8GB | || Qwen2.5-Coder | 7B | 4GB | || Phi-3 | 3.8B | 2GB | || Llama 3 | 70B | 36GB+ | |:--------:|:-----------:|:--------:| | 8GB | Qwen2.5:7B / Llama3.1:8B || | 16GB | Qwen2.5:14B / DeepSeek-Coder-V2:16B || | 32GB | Qwen2.5:32B / Command R:35B || | 64GB+ | Llama 3:70B / Qwen2.5:72B ||

VS Code + Continue.dev

{
  "models": [
    {
      "title": "Local Ollama",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b"
    }
  ]
}

docker run -d -p 3000:8080 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # 
)

response = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": ""}]
)
print(response.choices[0].message.content)

|:----:|:--------:|:--------:|---------|

# Ollama
export OLLAMA_NUM_PARALLEL=1

# 
# UseGGUFQ4

Summary

8GB
5Ollama + 

Use
100%