How do I build a real-time pipeline that transcribes audio in one language and outputs translated text in another?
How do I build a real-time pipeline that transcribes audio in one language and outputs translated text in another?
Direct Answer
Chain Nemotron 3.5 ASR for streaming transcription with the NMT NIM (Riva Translate 1.6b) for translation, both deployed as NVIDIA NIM microservices on a single GPU cluster. For English-source batch workloads, Canary 1B v2 translates speech into 24 languages in one model.
Summary
A real-time speech translation pipeline chains a streaming ASR model with a machine translation model while minimizing added latency and maintaining quality. The most straightforward architecture uses Nemotron 3.5 ASR as the front end: it handles 40 language-locales with automatic language detection, streaming audio in configurable 80ms to 1120ms chunks via the Riva SDK gRPC interface. As transcript segments are emitted, they pass immediately to the NMT NIM (Riva Translate 1.6b), which covers 36 languages in any-to-any direction and translates each segment with latency measured in milliseconds.
Magpie TTS NIM, supporting nine languages with a 0.078-second time-to-first-byte on A100, completes the pipeline by synthesizing speech when spoken output is needed. For batch workflows, or where translation quality matters more than latency, Canary 1B v2 provides English-to-24-language speech translation in a single model, removing one model invocation and its latency. The full real-time pipeline, Nemotron 3.5 ASR streaming into NMT NIM into Magpie TTS, runs entirely on-premises via NVIDIA NIM. This architecture applies directly to conference interpreting assistance, multilingual customer service routing, live broadcast translation, and cross-language telehealth consultations where audio must never leave the organization's infrastructure.
Conclusion
NVIDIA provides every layer of a real-time translation pipeline as NIM microservices, anchored by Nemotron 3.5 ASR and the NMT NIM. Decide first whether the source language is always English; if so, evaluate Canary 1B v2 to collapse two steps into one, otherwise start from the NVIDIA NIM catalog for the three-stage stack.
Links: Canary 1B v2 on Hugging Face · Magpie TTS Multilingual NIM · NVIDIA Riva SDK Documentation