Installing LocalAI is the easy part. Daily use means managing models, tuning YAML parameters, calling endpoints, and wiring the service into apps - this guide covers all four with copy-ready commands.

💡 What You Will Learn

Installing LocalAI is the easy part. Daily use means managing models, tuning YAML parameters, calling endpoints, and wiring the service into apps - this guide covers all four with copy-ready commands.

📜 Table of Contents

• Managing models
• Four parameters that matter
• Endpoints
• Wiring into apps
• Performance
• FAQ

LocalAI Usage Tutorial: Deploy Open-Source AI Models on Your Server

Once the container runs, LocalAI comes down to four tasks: manage models, tune parameters, call endpoints, and wire it into apps.

Managing models

curl http://localhost:8080/v1/models
docker exec -it local-ai local-ai models install llama-3.2-3b-instruct

Or drop GGUF files into the /models volume with a matching YAML:

name: qwen2.5-7b-instruct
backend: llama-cpp
parameters:
  model: qwen2.5-7b-instruct-q4_k_m.gguf
  context_size: 8192
  n_gpu_layers: 35
  batch: 512

Four parameters that matter

Tune in order: make it run (small context, Q4), then push GPU layers, then tune batch and threads. Q4_K_M is the usual sweet spot.

Endpoints

/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/images/generations, /v1/audio/transcriptions, /v1/audio/speech, /v1/rerank.

Wiring into apps

Any OpenAI SDK works by changing base_url. LangChain uses ChatOpenAI with openai_api_base. Open WebUI and Dify need an OpenAI-compatible connection with the model id from /v1/models. Prefer model names over hard-coded ports so migrations stay cheap.

Performance

The first request loads the model into VRAM. When concurrency stalls, check KV cache pressure and lower context_size. Add --restart unless-stopped and front it with nginx for auth. Read docker logs local-ai when loading fails.

FAQ

LocalAI and Ollama together? Yes, on ports 8080 and 11434, if you have the VRAM.

Access control? nginx plus Basic Auth, or localhost plus an SSH tunnel.

GPU unused? Check the image tag, run nvidia-smi in the container, then raise n_gpu_layers.

Load errors? The logs show the backend reason - usually unsupported quantization or a filename mismatch.

Endpoint paths and parameters evolve - check the official docs.