Local AI Assistant on Linux

A fully offline AI assistant on Linux: Ollama for models, Open WebUI for chat, LocalAI for OpenAI-compatible APIs.

A private assistant on a Linux box

Stack: Ollama (184,835 stars) serves models on port 11434, Open WebUI (156,799 stars) provides the browser interface with multi-user support and document Q&A, LocalAI (49,969 stars) is only needed when an application requires a strict OpenAI-compatible endpoint.

Install Ollama with the official script, then use systemctl edit ollama.service to set OLLAMA_HOST, OLLAMA_MODELS, OLLAMA_NUM_PARALLEL and OLLAMA_KEEP_ALIVE. Binding to 0.0.0.0 exposes an unauthenticated service, so keep it on localhost or put nginx basic auth in front.

Run Open WebUI from the official container image on port 3000 with a named volume for data, and point it at host.docker.internal:11434.

Three things decide whether it is really offline: pre-download every model including the embedding model, disable web search and external tools, and switch document Q&A to a local embedding model such as bge-m3. Then unplug the network and test.

Rough hardware: 8GB VRAM for 7B to 8B at Q4, 16GB for 14B, 24GB for 32B. Dropping quantization from Q8 to Q4_K_M is the first move when VRAM is tight.