Voice Assistant - Local Conversation Agent Sometimes Ignored

[rac146]

Describe the feature

Originally posted here:
home-assistant/core#162768

I have "Prefer handling commands locally" checked for my voice assistant:

When asking Assist a question that triggers a local sentence on the initial question, this works fine a expected (I know I triggered locally when I see 'Done'):

However, if you ask the wrong question and then try to correct yourself, the LLM seems stuck in Followup mode and doesn't look at the Local Agent again:

If you ask the LLM agent to RESET and then ask again, it (usually) works as intended:

I would expect every question to always look at the Local Agent first - there is a use case especially for voice where it might misunderstand your wording the first time and then you try to correct the voice input but then you get stuck in this state.

Example commands

See above..

Use cases

Example Use Case with Current Functionality:
(Assume a local trigger sentence exists for "close the bedroom curtains" )

User says: close the bedroom curtains
STT thinks: close the sadroom curtains
No trigger sentences match

User says again: close the bedroom curtains
STT: close the bedroom curtains
Even though there is a local trigger sentence it still won't match because this is an LLM followup?

Use case where "Prefer handling commands locally" ALWAYS looks at local trigger sentences first

User says: close the bedroom curtains
STT thinks: close the sadroom curtains
No trigger sentences match

User says again: close the bedroom curtains
STT: close the bedroom curtains
Trigger sentence matches

Anything else?

The "Prefer handling commands locally" option should ALWAYS prefer local commands first, even if it is a followup question. In a real-world scenario STT will never be 100% accurate. The current implementation is confusing where "Prefer handling commands locally" isn't always true.

[arturpragacz]

You could likely adjust your LLM prompt if you don't like that behaviour. Maybe simply adding something like "don't ask follow-up questions" would solve it for you.

[petep0p]

that doesn't sound like a good solution. that would eliminate the follow-up feature entirely, when that is clearly not the desired outcome. The feature request is to always check against sentence intents for a user input regardless of the follow-up status. to get around the issue where the LLM might be stuck waiting for the follow-up response, there could be a mechanism to send a cancel/stop message to the LLM when this case (where the follow-up response is replaced by a sentence intent match) comes up. but we would generally want the follow-up feature to persist for those cases where we actually want that behavior.

[mitrokun]

I implemented a hack to bypass this problem for myself.
If you're interested, you can test my integration (use the specified branch) or use the code for your own needs.
I agree that blocking local intents while a continued conversation is in progress is incorrect.