Home Assistant with a Local LLM: Voice Control without the Cloud

Home Assistant’s built-in voice control works on the principle of full-text patterns: there is a set of templates for every command, and anything that does not fit a template is not understood. “Turn on the light in the office” works. “It is too dark to work in the office” does not, unless that exact phrasing has previously been added as a custom intent.

Most people know the alternative from consumer devices: Google Assistant or Alexa understand free-form language, but every command leaves the local network and the device inventory lives with a vendor. That is exactly what I did not want. Fortunately, Home Assistant’s Conversation interface has become a real pipeline: STT in, conversation agent in the middle, TTS out, with every stage replaceable. The agent in the middle does not have to be OpenAI; it only has to speak the same API.

My goal was free-form German commands and follow-up questions against my Home Assistant installation, completely local. “Is a window still open?”, “Make it bright in the office” or “Turn down the heating in the bedroom if nobody is there”, without a single syllable leaving the house.

The setup

The chain looks like this: Home Assistant runs as before. The OpenAI integration is configured as the conversation agent, but instead of pointing to OpenAI it points at the OpenAI-compatible endpoint of my Ollama server. This is the same server that also handles document classification for Paperless. For voice, Whisper for speech recognition and Piper for speech output are connected to Home Assistant through the Wyoming protocol, while an ESP32-S3 with Micro Wake Word serves as the satellite on my desk.

The important part is that the text path needs none of those voice components. Anyone who only writes through the Assist chat box or the app is done with Home Assistant plus Ollama. The voice pipeline is an optional second project and, as it turned out, the considerably more involved one.

Connecting the Conversation agent to Ollama

The OpenAI integration needs three fields:

Basis-URL: http://<ollama-server>:11434/v1
API-Key:   ollama            # beliebiger Platzhalter, Ollama prüft ihn nicht
Modell:    qwen2.5:7b-instruct

Ollama has offered a /v1 endpoint that emulates the OpenAI Chat Completions API for quite some time. Home Assistant does not notice the difference. The integration requires an API key, but Ollama ignores it, so a placeholder is enough.

The decisive setting is “Control Home Assistant” on the agent. With it enabled, the model receives the list of exposed entities and their current states with every request and is allowed to call the corresponding tools: switch devices, query states and activate scenes. Without that option it is a chatbot that can talk about my house but cannot do anything in it.

That also defines the model requirement: the model has to support function calling. This was the first dead end, which I will get to in a moment. I use a 7B model from the Qwen2.5 family; Llama 3.1 models with tool support work as well. This is not a benchmark recommendation, just the result of some experimentation. Anyone reproducing the setup should test the model already available first and only switch when tool calls are unreliable.

What the LLM can do that intents could not

The traditional way to add custom commands to Home Assistant is intent_script. For every sentence the house is supposed to understand, the intent and action are written explicitly:

intent_script:
  BueroHell:
    action: light.turn_on
    target:
      entity_id: light.buero
    speech:
      text: "Büro ist jetzt hell."

That is reliable and deterministic. It simply does not scale with the creativity of the things people actually say. Every wording, every combination and every follow-up becomes another entry.

The LLM removes three constraints. First, strict wording disappears: “Make it bright in the office”, “Office light on” and “I cannot see anything” all end up at the same action because the model matches intent rather than the literal words. Second, context can span several sentences. After “Is a window still open?”, the follow-up “And the door?” is understood as referring to the door sensors. Third, and this is the real gain, it can answer queries across states for which no intent exists at all. “Is a window still open?” is not a switching command but an aggregation across all window contacts. I never programmed that aggregation; the model reads the state list and answers.

The agent can also be called directly from automations, for example to generate an announcement:

action: conversation.process
data:
  agent_id: <conversation-agent-id>
  text: "Fasse zusammen, welche Fenster offen sind, in einem Satz."
response_variable: ansage

The response is available in ansage.response.speech.plain.speech and can be sent to Piper or a media player. Announcements are then formulated dynamically instead of playing predefined strings.

Three dead ends along the way

The first dead end was a model without function calling. It sounded convincing and said things such as “I turned on the light”, but nothing had been switched. Some models wrote the tool call as JSON into the answer text, which Home Assistant then read aloud to the user. Without proper tool support in the model, the entire integration is decoration. This takes a minute to test: configure the agent, send “Turn on the living-room lamp” and check whether the log contains a tool call. If it does not, change the model rather than tweaking the prompt.

The second dead end was my own ambition. I initially exposed everything, almost three hundred entities. The result was a large context on every request, noticeable latency, and a model that struggled to choose between similarly named lights and occasionally switched the wrong device. The solution is a short allowlist containing only the devices the assistant actually needs to control, with clear names and aliases.

The third dead end was speech recognition. Short German commands are an unforgiving case for small Whisper models: little context, many proper names, and a model that may hallucinate English when uncertain. “Flurlicht aus” has, for example, turned into an English-sounding phrase and then into a question about houses. Two measures helped: pin the language to German so the model does not guess again whenever there is silence, and choose a model size the hardware can handle instead of automatically picking the smallest variant.

Voice is the expensive part

After building the system, my conclusion is clear: the LLM agent was the quick part. Configure the integration, curate the allowlist, done. Text chat worked the same evening.

The voice pipeline took the evenings after that. Micro Wake Word on the ESP32 works, but it is sensitive to room acoustics and the placement of the satellite. The German Piper voices sound like a railway-station announcement: understandable, but nobody will confuse them with a cloud assistant. And every stage of the chain adds latency: wake up, record, transcribe, think, answer, speak. It is noticeably slower than a silent switch. That is my subjective assessment, not a measurement. Anyone expecting it to feel like a polished commercial product will be disappointed.

To be fair, that latency and synthetic voice are the price of not pointing a microphone in the house at somebody else’s server.

Conclusion

What runs in the end is free-form German control against a curated device list, state queries without predefined intents, and dynamically written announcements from automations. Everything is local and works without an account at a provider that might change the API next quarter. I no longer underestimate that last point: a self-hosted open-weight model does not change behaviour overnight because a vendor deploys an update or retires a model. What works today will still work the same way tomorrow. It is the same basic philosophy as the automations that continue working without internet access.

The cost is a machine capable of running the model, in my case the same Ollama server used for Paperless, noticeable voice latency, and the acceptance that a 7B model is worse at switching devices than a cloud assistant backed by years of intent training. Wrong actions happen. That is why the allowlist is short and why anything security-critical stays out of reach of the model.

For me the trade-off still works. I no longer maintain sentence templates, the state queries are genuinely useful, and I retain control over the entire pipeline. Anyone who does not need a voice satellite gets the main benefit, the agent that actually understands free-form language, with a fraction of the effort: Home Assistant, an Ollama endpoint and a short allowlist. The rest is luxury and tinkering.