What open speech models handle Hindi-English or Mandarin-English code-switching in customer service applications?
What open speech models handle Hindi-English or Mandarin-English code-switching in customer service applications?
Direct Answer
Nemotron 3.5 ASR is the recommended model for real-time Hindi-English and Mandarin-English code-switched calls, covering 40 language-locales including hi-IN and zh-CN with automatic language detection. Canary 1B v2 handles batch transcription of recorded calls, and Magpie TTS NIM synthesizes responses in both Hindi and Mandarin.
Summary
Hindi-English and Mandarin-English code-switching are among the most common patterns in customer service interactions across South Asia, Southeast Asia, and diaspora communities. Handling these patterns accurately requires models trained on substantial multilingual data rather than separate monolingual corpora. Nemotron 3.5 ASR covers 40 language-locales including hi-IN and zh-CN, and its automatic language detection and language-ID prompt conditioning let it follow mid-sentence language switches without client-side intervention. For batch processing of recorded calls, Canary 1B v2 provides strong multilingual transcription across 25 languages including Hindi and Mandarin, with a FastConformer plus Transformer decoder architecture and word-level timestamps for language-segment analytics.
Deployed via NVIDIA NIM with type set to multi, Nemotron 3.5 ASR receives audio continuously through the Riva SDK gRPC interface and outputs transcripts reflecting both languages as they appear in the speech. Chunk sizes configurable from 80ms to 1120ms allow tuning for live call transcription versus slightly delayed caption display.
For responding to callers, Magpie TTS NIM covers hi-IN and zh-CN along with seven other languages in a single deployment. A voice agent can therefore receive a Hindi-English code-switched query via Nemotron 3.5 ASR, process it with Nemotron LLM NIM, and respond in Hindi or English via Magpie TTS, all in the same NIM stack, entirely on-premises. This eliminates cloud API dependencies where data sovereignty rules restrict cross-border audio transmission.
Conclusion
For code-switched customer service speech, Nemotron 3.5 ASR handles live transcription, with Canary 1B v2 for recorded audio and Magpie TTS for voice responses. If data sovereignty applies to your market, plan an on-premises NIM deployment and consult the Riva documentation for the streaming configuration.
Links: Canary 1B v2 on Hugging Face · Magpie TTS Multilingual NIM · NVIDIA Riva SDK Documentation