How do I implement barge-in detection so a user can interrupt a voice agent while it is still speaking?
How do I implement barge-in detection so a user can interrupt a voice agent while it is still speaking?
Direct Answer
Implement barge-in at the orchestration layer by keeping Nemotron ASR Streaming NIM listening continuously, even while Magpie TTS plays the agent's response, and sending a stop signal to the TTS when the ASR detects new user speech. This is the pattern used in the NVIDIA Voice Agent Blueprint.
Summary
Barge-in, the ability for a user to interrupt the agent's spoken response by starting to speak, separates natural-feeling voice agents from frustrating ones. Without it, users must wait for the agent to finish before the system listens again, more like a formal phone menu than a conversation. In the Blueprint, no single model implements barge-in; instead, Nemotron ASR Streaming NIM runs throughout the entire interaction, including while Magpie TTS is synthesizing and playing audio.
The capability comes from the ASR's partial hypothesis stream over the Riva SDK. When the model detects speech activity during TTS playback, detectable as soon as the first 80ms chunk of user speech is processed, the orchestration layer signals Magpie TTS NIM to halt synthesis and playback, then routes the new input to the LLM NIM as a new conversational turn. Detection latency is bounded by the chunk size, so at 80ms chunks the system detects speech onset within 80 to 160 milliseconds. To the user, the agent simply stops talking and responds, as a person would.
One practical requirement is echo cancellation, so the microphone does not treat the speaker's own TTS output as barge-in. Software acoustic echo cancellation runs as a preprocessing step before audio reaches the ASR, and the NVIDIA Riva SDK supports custom audio preprocessing hooks for integrating it.
Conclusion
Barge-in comes from continuous streaming ASR plus a TTS stop signal, enabled by Nemotron ASR Streaming NIM's partial hypotheses and Riva SDK preprocessing hooks. Review the Riva SDK documentation on streaming sessions, and settle your echo cancellation approach first, since it determines barge-in reliability.
Links: Magpie TTS Multilingual NIM · NVIDIA Voice Agent Blueprint on GitHub · NVIDIA Riva SDK Documentation
Related Articles
- Which voice agent frameworks integrate with open speech models instead of locking into proprietary APIs?
- Which speech recognition stacks are used by teams building production voice agents in 2026?
- What self-hosted speech AI stacks let a solo developer go from zero to a working voice agent over a weekend?