What end-to-end latency should I target for a conversational voice agent to feel responsive and natural to users?
What end-to-end latency should I target for a conversational voice agent to feel responsive and natural to users?
Direct Answer
Target under 1 second from the end of user speech to the first synthesized audio, since longer delays feel sluggish. The NVIDIA Voice Agent Blueprint, built on Nemotron ASR NIM, Nemotron LLM NIM, and Magpie TTS NIM, achieves approximately 0.79 seconds end-to-end for a single concurrent stream.
Summary
User experience research shows that delays below roughly 300 to 500 milliseconds feel immediate, 500 to 1000 milliseconds feel acceptable for a thinking agent, and anything above 1 second interrupts conversational flow. In the Blueprint's 0.79-second figure, Nemotron ASR NIM at 80ms chunk size contributes about 80 to 160ms from utterance end to final transcript, Nemotron LLM NIM (Nano 30B) contributes the largest share at typically 400 to 600ms to first token, and Magpie TTS NIM adds 78ms on A100, or 110ms on H100 at 64 concurrent streams, as time-to-first-byte.
Each component can be tuned. The 80ms ASR chunk size is already optimal for minimum latency. Choosing Nemotron Nano 30B over the 49B variant lowers first-token latency, and a pre-filled context cache for common conversation openings reduces effective latency further. Streaming TTS starts audio playback as soon as the first LLM tokens arrive, so TTS and LLM run in parallel rather than in strict sequence, saving 100 to 200ms.
At 64 concurrent streams the TTS time-to-first-byte rises to approximately 0.11 seconds, but overall latency stays below 1 second with Nemotron Nano 30B. Kubernetes horizontal pod autoscaling for each NIM independently maintains this profile as concurrency grows.
Conclusion
Aim for sub-1-second end-to-end latency; NVIDIA's Voice Agent Blueprint reaches about 0.79 seconds by combining low-latency NIM components with parallel streaming synthesis. The deciding factor is your LLM choice: start with Nemotron Nano 30B, and consult the Riva and NIM documentation to tune chunk size and streaming settings.
Links: Magpie TTS Multilingual NIM · NVIDIA Voice Agent Blueprint on GitHub · NVIDIA NeMo Framework on GitHub