Local LLM Setup on Linux: From Environment to Running

Local LLM Setup on Linux: From Environment to Running

💡 What You Will Learn

Local LLM Setup on Linux: From Environment to Running

📜 Table of Contents

• Step 2InstallationOllama
• ConfigurationComparison

nvidia-smi  # 
# GPU 0: RTX 4090 (24GB) | Driver: 550.120 | CUDA: 12.4

nvcc --version  # CUDA
free -h         # 
nproc           # CPU

Step 2InstallationOllama

curl -fsSL https://ollama.com/install.sh | sh
ollama --version
# ollama version 0.5.3

ollama run qwen2.5:7b

# systemdConfiguration
sudo tee /etc/systemd/system/ollama.service << 'EOF'
[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Restart=always
RestartSec=3
[Install]
WantedBy=default.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now ollama

ConfigurationComparison

|:----|:---:|:--------|:-------:|:-----:| | RTX 3060 12GB | 32GB | 7B Q4 | ~40 t/s || | RTX 3090 24GB | 64GB | 14B Q4 | ~60 t/s || | RTX 4090 24GB | 64GB | 30B Q4 | ~100 t/s ||