Home Assistant (89,719 stars) now has native LLM support: connect an Ollama model to control your smart home by voice or chat - fully private, no cloud, no monthly fee.

💡 What You Will Learn

Home Assistant (89,719 stars) now has native LLM support: connect an Ollama model to control your smart home by voice or chat - fully private, no cloud, no monthly fee.

📜 Table of Contents

• The chain
• Step 1: Model
• Step 2: Add the integration
• Step 3: Expose entities, then restrict
• Step 4: Build the pipeline
• Step 5: Tune the prompt
• What to expect
• FAQ

Home Assistant Meets a Local LLM: A Full Ollama Voice Assistant Setup

Most smart-home voice assistants upload audio to the cloud. Home Assistant has shipped LLM conversation support since 2024, and with a local Ollama model the whole voice chain can stay inside your network.

The chain

Mic → STT (Whisper) → conversation agent (Ollama) → intent execution (HA)
                                ↓
                        TTS (Piper) → speaker

Every link can run locally: Whisper transcribes, Ollama reasons and calls tools, Piper speaks, HA actuates.

Step 1: Model

Agents need reliable structured commands more than eloquence.

ollama pull qwen3:8b
ollama pull qwen3:4b

For lights and sensors a 4B model is usually enough and noticeably faster. Verify the service:

curl http://<ollama-host>:11434/api/tags

If HA runs elsewhere, make sure Ollama listens on the LAN interface, not just loopback.

Step 2: Add the integration

Option A — official Ollama integration (recommended): Settings → Devices & Services → Add Integration → Ollama → address http://<ollama-host>:11434 → pick a model.

Option B — OpenAI Conversation: point the Base URL at http://<ollama-host>:11434/v1 and enter any non-empty key. Useful when your model is not listed or you want one protocol everywhere.

Both expose a "control Home Assistant" toggle, which is what lets the model actually actuate devices. It is off by default.

Step 3: Expose entities, then restrict

1. Settings → Voice assistants → Expose entities, tick only what should be controllable.
2. Do not expose everything. Garage doors, locks and cameras stay hidden by default.
3. Give devices clear aliases — "Living room main light" matches far better than light.yeelight_ceiling_3.

Fewer visible entities means fewer wrong actions, shorter prompts and faster responses. Roll out room by room.

Step 4: Build the pipeline

Settings → Voice assistants → Add assistant, four stages:

Wyoming is HA's protocol for local voice services; Whisper and Piper both have container images you can host on the same internal server.

Step 5: Tune the prompt

Typical problems: verbose answers, treating chit-chat as commands, mixed-language output.

You are a home assistant. Answer only questions about controlling home devices.
Keep answers under two sentences, use English only, no emoji.
If a device name is ambiguous, ask instead of guessing.

What to expect

• Latency: STT plus inference dominates; a few seconds end to end is normal.
• Success rate: simple commands work well with clean entity names; compound commands often drop the second half.
• Offline: with everything local, devices still respond with the internet down — the main reason to do this.

FAQ

Q: The model toggles the wrong device. Narrow the exposure list and check for duplicate or vague aliases — the most common cause.

Q: Cloud models? Supported via API key with better quality, but audio and commands leave the network.

Q: Poor Chinese STT? Use a larger Whisper model; undersized models are the usual culprit.

Q: Special hardware? ESPHome voice satellites are best, but a phone or browser proves the flow before you buy anything.

Q: HomeKit? HA bridges devices to HomeKit, so local assistant control carries over.

Q: Context memory? Works, but long context slows responses and can leak a previous device name into a new command.

Integration names and options change between HA releases; check official docs.

❓ FAQ

Can the LLM control devices safely?

Yes - HA exposes allowed entities only, and you can restrict which devices the model can touch in the integration config.

What if I want cloud models?

The same integration accepts OpenAI or Anthropic API keys - but then voice audio and commands go to the cloud.

Does it work with Apple HomeKit?

Yes - HA bridges to HomeKit, so the LLM assistant can control HomeKit devices too.