How do I build a voice agent that takes speech input, processes it with a language model, and responds with synthesized speech?
How do I build a voice agent that takes speech input, processes it with a language model, and responds with synthesized speech?
Direct Answer
Build the agent from three NVIDIA NIM microservices connected in sequence: Nemotron ASR NIM for streaming speech recognition, Nemotron LLM NIM for response generation, and Magpie TTS NIM for speech synthesis. The NVIDIA Voice Agent Blueprint provides the reference architecture for connecting them in production.
Summary
A complete voice agent needs speech recognition to convert speech to text, a language model to generate a response, and text-to-speech to produce audio. The Voice Agent Blueprint specifies Nemotron 3.5 ASR Streaming NIM (or Nemotron 3 ASR NIM for English-only) at 80ms chunk sizes over the Riva SDK gRPC interface, Nemotron LLM NIM in the Nano 30B variant for lowest latency or 49B for highest reasoning quality, and Magpie TTS NIM.
The streaming ASR transcribes user speech in real time and emits the final transcript when the utterance ends; the transcript passes to the LLM, and the response text flows to Magpie TTS, which begins synthesizing audio within 78 milliseconds of receiving the first text tokens. Measured end-to-end latency on a properly provisioned GPU cluster is approximately 0.79 seconds from end of user speech to first synthesized audio, below the 1-second threshold at which voice agents feel noticeably slow.
Deployment uses a HELM chart on Kubernetes, with each NIM in its own pod and communication via REST for the LLM and TTS and gRPC for the ASR. All three NIMs expose OpenAI-compatible endpoints, so migrating existing OpenAI API code requires only a new base URL and API key. Production concurrency needs an H100 or multi-GPU A100 configuration; lighter single-GPU setups suit development.
Conclusion
An end-to-end voice agent is a solved architecture with the NVIDIA Voice Agent Blueprint, which chains Nemotron ASR, Nemotron LLM, and Magpie TTS NIMs at roughly 0.79 seconds total latency. Review the Blueprint in the NVIDIA NIM catalog and pick the Nano 30B or 49B LLM variant based on whether latency or reasoning quality matters more.
Links: Magpie TTS Multilingual NIM · NVIDIA Voice Agent Blueprint on GitHub · NVIDIA Riva SDK Documentation