Mixed language voice pipeline

[zolakt]

Describe the feature

This is a novelty item, but I think a lot of non-english users would like this. Voice assistant is heavily centered around english (and possibly a few major languages), but outside of that, options are limited.

Would it be possible to have the request and responses be in a different language, at least when using an LLM pipeline? I don't want to talk to my assistant in english. I talk to it in croatian. However there are only 2 TTS voices for croatian, and both are dull.

It would be really cool to give a command in croatian, and get the response in english, which has incomparably more TTS options.

Example commands

Request: "Hey Jarvis, upali svjetlo u kuhinji"
Response: "Light turned on"

Use cases

I would get an incomparably better TTS experience than listening to the dull l, robotic, croatian voice, e.g. the GLaDOS TTS.

The choice of TTS voices is going to expand eventually, and while english speakers will be using things like darth vader, mortal kombat announcer etc, I'll be stuck with the same 2 dull croatian voices

Anything else?

No response

[mitrokun]

Prepare the desired voice dataset and train Piper. It's quite simple, any AI will explain all the steps. You can use Google Colab for training if you don't have the right equipment.
https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/TRAINING.md

[zolakt]

That doesn't really solve my issue. I'd like it to respond in a different language, where I have more options.

Also I don't want to train it with my voice, and then talk to myself. I find that just weird and narcissistic

[mitrokun]

I'll be stuck with the same 2 dull croatian voices

I am not offering a voiceclone of your voice. Prepare data for textured voices, such as Croatian announcers and dubbing actors.
This is the correct solution to the problem.

As for the main request, although it is very specific, you can implement responses in English, both for built-in commands (requires replacement of response files) and AI (the instructions must contain clear instructions about the language). You will also need to modify the voice server so that it ignores the language in the request and always processes English. If you use F5 and similar single-voice TTS, this is very simple.
There is a reverse option, but only for interaction via AI. You select the English pipeline and use multilingual ASR, which determines the language itself. For example, nemo-parakeet-tdt-0.6b-v3. This is a more limited use case that requires a very fast LLM.

[zolakt]

I am not offering a voiceclone of your voice. Prepare data for textured voices, such as Croatian announcers and dubbing actors.
This is the correct solution to the problem.

This is a solution to get a voice that works locally (since there are none currently, only cloud options), but it doesn't address what I'm proposing. Even if I get someone to record all the traning data, it will be a single voice. It's unrealistic to expect Croatian will ever catch up with English in terms of different options. And the possibility that there will ever be some more interesting voices in Croatian (e.g. morgan freeman clone or something) is zero.

As for the main request, although it is very specific, you can implement responses in English, both for built-in commands (requires replacement of response files) and AI (the instructions must contain clear instructions about the language). You will also need to modify the voice server so that it ignores the language in the request and always processes English. If you use F5 and similar single-voice TTS, this is very simple.
There is a reverse option, but only for interaction via AI. You select the English pipeline and use multilingual ASR, which determines the language itself. For example, nemo-parakeet-tdt-0.6b-v3. This is a more limited use case that requires a very fast LLM.

While these workaround may possible work, they are basically monkey-patches. I really don't want to replace all the response files, and then after every update go through all of them to see if something was added/changed. Or modify the server manually, which will get overwriten with the next update. As for the AI case, yeah, that could possibly work, although autodetecting the ASR language makes it much much slower. Not being locked into a single language in the pipeline would be a much more elegant solution.

[bexelbie]

I don't have an answer, but I believe a core challenge is that the current Voice Assist pipelines only handle one language, even if you tell your STT to use more than one. I think there is a config passed in that overrides the STT's native multi-language support.

[webysther]

I want to give more details here:

In my home is speak 3 languages: portuguese of brazil, italian and english. We mix all languages and working with IT for instance normally I have a zone of my house as "Hall" and not "Corredor", If I ask starting in portuguese and add english or italian names the STT make a mess, no matter if is a LLM because the only model that support multiple languages is whisper-1 on openai api today, which don't work.

The assist works great, but is not prepared for multiple languages. One option that is fundamental needed is the option to select "Any" or a list of languages.

[webysther]

Related to https://community.home-assistant.io/t/allow-multiple-languages-in-ha-assist/535030