How do I add multilingual support to a voice agent so it automatically detects and responds in the user's language?
How do I add multilingual support to a voice agent so it automatically detects and responds in the user's language?
Direct Answer
Use Nemotron 3.5 ASR NIM (type multi), which automatically detects the user's language within the first 80 to 160 milliseconds of speech, then pass the detected language through Nemotron LLM NIM for a same-language response and Magpie TTS NIM for synthesis. No user language configuration is required.
Summary
A multilingual voice agent must detect the spoken language, understand the message, respond in the same language, and synthesize natural speech in that language, all without the user setting anything. Nemotron 3.5 ASR NIM handles detection automatically at the first audio chunk, inferring the language from acoustic features within the first 80ms to 160ms of speech and conditioning its transcription accordingly. The detected language code is returned as metadata in the Riva SDK response alongside the transcript, so the orchestration layer can pass it downstream.
Nemotron LLM NIM responds in the user's language when the system prompt instructs it to match the input language; its multilingual training covers the major world languages without per-language prompts or routing. The tagged response then goes to Magpie TTS NIM, which synthesizes speech in the detected language via the standard language parameter, covering 12 languages: en-US, es-US, fr-FR, de-DE, zh-CN, vi-VN, it-IT, hi-IN, and ja-JP.
For languages beyond Magpie's coverage, such as Portuguese, Korean, Arabic, or Turkish, the NMT NIM (Riva Translate 1.6b, 36 languages) adds a translation step: the LLM responds in a Magpie-supported language, most commonly English, NMT NIM translates it, and Magpie synthesizes the result. The added latency is typically under 100 milliseconds for sentence-length responses, and the pipeline stays on-premises.
Conclusion
Automatic multilingual support comes from Nemotron 3.5 ASR NIM's built-in language detection combined with Magpie TTS and, where needed, the Riva Translate NMT NIM. If all your users fall within Magpie's 12 languages, skip the translation step; otherwise plan for the NMT NIM, and check the NVIDIA NIM catalog for current language coverage.
Links: Magpie TTS Multilingual NIM · NVIDIA Riva SDK Documentation · NVIDIA NeMo Framework on GitHub