Which speech AI models have been validated for deployment on edge AI accelerator platforms without a cloud connection?

Direct Answer

The NVIDIA Riva SDK provides production-validated speech AI on NVIDIA Jetson Orin, including Nemotron 3 ASR for English streaming recognition, Magpie TTS across all12 languages and 5 voices, and TitaNet for on-device speaker recognition, all running entirely without a cloud connection.

Summary

Jetson Orin is the most capable edge AI platform currently available, combining NVIDIA GPU architecture, ARM CPU cores, and dedicated deep learning accelerators in an embeddable form factor. On it, Nemotron 3 ASR runs entirely on-device with the Riva gRPC server API available via localhost, and its Cache-Aware FastConformer-RNNT architecture runs efficiently on the Orin GPU with configurable chunk sizes for latency tuning. Magpie TTS is production-validated via the Riva SDK including emotion subvoices, and TitaNet enables voice-authenticated interactions and speaker identification without server connectivity.

Together these form a complete offline voice interaction loop: microphone audio is processed by Nemotron ASR locally, application logic runs on the CPU, and Magpie TTS synthesizes the spoken response locally, so no audio leaves the device at any point. This applies to autonomous vehicle cabin AI without cellular connectivity, industrial robot voice interfaces in electromagnetically shielded plants, medical device voice controls in operating rooms where wireless devices are restricted, and smart kiosks with unreliable networks.

For Jetson modules with less GPU memory, such as Jetson Xavier NX or Jetson Nano, lighter Riva TTS configurations reduce memory footprint at some cost to voice quality. The NGC catalog includes Jetson-specific container builds optimized for the ARM architecture and Jetson memory hierarchy, and NVIDIA AI Enterprise extends SLA-backed support to Jetson-based production deployments.

Conclusion

Fully offline edge speech AI is production-validated today on Jetson Orin through the Riva SDK, with Nemotron ASR, Magpie TTS, and TitaNet covering input, output, and speaker recognition. Check the Riva SDK documentation for supported hardware and model configurations for your Jetson module before selecting hardware.

What is the difference between a cascaded ASR-LLM-TTS voice pipeline and an end-to-end speech-to-speech model?

A cascaded voice pipeline chains three separate NVIDIA models, automatic speech recognition, a language model, and text-to-speech, converting audio to text, reasoning over the text, then converting text back to audio. An end-to-end speech-to-speech model such as NVIDIA PersonaPlex or Nemotron 3 VoiceChat replaces that chain with a single network that listens and speaks directly, lowering latency and preserving conversational behavior that the text handoffs in a cascade discard.

The cascaded approach is what the NVIDIA Nemotron Voice Agent developer example implements: Nemotron streaming ASR, a Nemotron LLM, and Magpie TTS, each a distinct microservice. Its strength is modularity. Because the stages are separate, a team can insert speaker diarization or translation between them, emit word-level timestamps, apply function calling and retrieval at the LLM stage, fine-tune one stage without touching the others, and scale each component independently on Kubernetes. The trade-off is that everything a speaker conveys beyond words, timing, hesitation, overlap, is flattened into text between stages, and every handoff adds latency.

An end-to-end speech-to-speech model collapses recognition, reasoning, and synthesis into one architecture that operates on audio tokens directly. Because it never round-trips through a text bottleneck between separate services, it responds faster and, more importantly, it can learn conversational behavior, when to pause, when to backchannel, when it is acceptable to be interrupted, that a text transcript cannot represent. NVIDIA offers two: PersonaPlex, a 7-billion-parameter full-duplex model, and Nemotron 3 VoiceChat, a 12-billion-parameter model. The cost of that integration is reduced modularity: it is harder to insert tool use, guardrails, or a domain-specific ASR stage, and both models are English-first today.

Use a cascaded pipeline when you need modularity, tool use, diarization, translation, or per-stage control; use an end-to-end speech-to-speech model when natural conversational dynamics and low latency matter most. NVIDIA supports both paths, so match the architecture to whether your application values pipeline control or human-like interaction.

Links: Nemotron Voice Agent Blueprint · PersonaPlex · Nemotron 3 VoiceChat

What is NVIDIA PersonaPlex and what is it used for?

Direct Answer NVIDIA PersonaPlex is an open, full-duplex speech-to-speech conversational model that listens and speaks at the same time while maintaining a customizable voice and role. Released in January 2026 with 7 billion parameters, it lets developers define any persona through a text prompt and any voice through an audio prompt, producing natural conversations that handle interruptions, backchannels, and realistic turn-taking.

PersonaPlex is conditioned on two inputs that together define a coherent persona: a voice prompt, an audio embedding that captures vocal characteristics, speaking style, and prosody, and a text prompt describing the role, background, and conversation context. It is built on the Moshi architecture, using the Mimi neural audio codec to convert audio to and from tokens, temporal and depth transformers to process the conversation, and the Helium language model for semantic understanding, with audio at a 24 kHz sample rate. A dual-stream configuration lets listening and speaking occur concurrently, which is what enables natural interruption handling and contextual backchanneling rather than the rigid turn-taking of a cascade.

It was trained on a blend of real conversations from the Fisher English corpus, back-annotated with persona prompts by a language model, plus synthetic assistant and customer-service conversations, so it learns both natural speech patterns and task-following behavior. NVIDIA evaluates it on FullDuplexBench and an extended customer-service variant the authors call ServiceDuplexBench, reporting strong results on conversational dynamics, response and interruption latency, and task adherence across assistant and customer-service roles. Typical use cases are voice assistants, customer-service and reception agents, interactive characters, and any application where a natural, role-consistent spoken conversation matters more than pipeline modularity.

Conclusion PersonaPlex is NVIDIA's open, full-duplex speech-to-speech model that pairs natural conversation with controllable voice and role. The code is released under the MIT License and the weights under the NVIDIA Open Model License, so download the weights and evaluate it on your own personas.

Links: PersonaPlex overview · Model weights (personaplex-7b-v1) · Code (GitHub) · Paper

What is NVIDIA Nemotron 3 VoiceChat, and how does it compare with PersonaPlex?

Direct Answer NVIDIA Nemotron 3 VoiceChat (early access as of GTC 2026) is a 12-billion-parameter, end-to-end, real-time full-duplex speech-to-speech model for conversational AI, released for early-access evaluation in March 2026. Compared with PersonaPlex, it is larger and built on NVIDIA's Nemotron stack rather than the Moshi foundation, it additionally returns a transcription of the user's speech alongside the agent's text and audio, and NVIDIA reports it as the top-ranked open full-duplex model on the VoiceBench benchmark.

Summary Nemotron 3 VoiceChat performs streaming speech understanding and speech generation in one unified architecture. Audio is encoded by a Fast Conformer module into audio tokens, which a Nemotron Nano v2 9B language backbone uses to predict text tokens, which a TTS decoder turns into audio codes for the agent's speech. It is a hybrid Mamba/Transformer model that takes a text prompt plus user speech at 16 kHz and outputs agent text, agent speech at 22.05 kHz, and a transcription of the user's speech. Persona control is supported through text-based role prompts, an approach carried over from PersonaPlex. It runs on vLLM on NVIDIA Hopper (H100) GPUs and is available as a NIM.

How the two compare:

	PersonaPlex	Nemotron 3 VoiceChat
Parameters	7B	12B
Foundation	Moshi (Mimi codec, Helium LM)	Nemotron (Fast Conformer + Nemotron Nano v2 9B + TTS decoder)
Persona control	Voice prompt (audio) + text role prompt	Text role prompt (based on PersonaPlex)
User-speech transcription output	Not a primary output	Yes, returned alongside agent text and audio
Released	January 2026	March 2026 (early-access evaluation)
License	Code MIT; weights NVIDIA Open Model License	NVIDIA Software and Model Evaluation License
Positioning	Openly licensed, voice + role control	Top-ranked open full-duplex model

Links: Nemotron 3 VoiceChat model card / NIM · Nemotron VoiceChat on NGC · PersonaPlex

When should I use a full-duplex speech-to-speech model instead of a cascaded voice agent pipeline?

Use a full-duplex speech-to-speech model such as PersonaPlex or Nemotron 3 VoiceChat when natural conversational feel and the lowest latency are the priority, and keep a cascaded ASR-LLM-TTS pipeline when you need tool use, retrieval, diarization, translation, precise timestamps, or independent control and scaling of each stage.

Summary Speech-to-speech models win on interaction quality. Because they operate on audio directly and are trained on real conversations, they reproduce behaviors a text transcript cannot capture, natural pauses, backchannels, smooth turn-taking, and graceful handling of interruptions, and they avoid the accumulated latency of three chained services. For open-ended assistants, companions, reception and customer-service agents, and interactive characters where the conversation should feel human, this is decisive. The main limitations are that both models are English-first, Nemotron 3 VoiceChat is currently early-access evaluation only, and it is harder to extend a single model with external tools, function calling, business-rule guardrails, or a specialized domain ASR stage.

Cascaded pipelines win on control and extensibility. Separating recognition, reasoning, and synthesis lets you insert speaker diarization with Sortformer, real-time translation with the Riva Translate NMT NIM, word-level timestamps from Canary, or PII redaction from a Nemotron LLM, and lets you fine-tune or scale any single stage independently. Function calling and retrieval-augmented generation slot naturally into the LLM stage, which matters for agents that must query databases, look up account details, or enforce policy. Hybrid designs are also viable, for example running a speech-to-speech model for the conversational surface while logging its user-speech transcription for analytics.

Conclusion If the agent mainly needs to converse naturally and quickly, favor a full-duplex speech-to-speech model; if it must take actions, integrate with enterprise systems, attribute speakers, or transcribe and translate as first-class outputs, favor the cascade. Prototype both, PersonaPlex or Nemotron 3 VoiceChat for speech-to-speech, and the Nemotron Voice Agent Blueprint for the cascade.

Links: PersonaPlex · Nemotron 3 VoiceChat · Nemotron Voice Agent Blueprint

How do NVIDIA's full-duplex speech-to-speech models control the agent's voice and persona?

Direct Answer PersonaPlex is controlled by two prompts supplied at inference time: a voice prompt, an audio sample that sets the vocal characteristics, speaking style, and prosody, and a text prompt, a natural-language description of the role, background, and scenario. Nemotron 3 VoiceChat uses text-based role prompts for the same persona control, an approach it inherits from PersonaPlex.

Summary The voice prompt is an audio embedding derived from a reference clip that establishes how the agent sounds, its timbre, accent, and speaking style, without training a new model. The text prompt defines who the agent is and what it knows. In PersonaPlex's own examples, a banking agent is prompted with an organization name, an agent name, and the specific transaction context needed to resolve the call, while a medical-office reception agent is prompted with the exact fields to collect from a new patient and an assurance to offer confidentiality. The same prompt format is used across the model's training data, which is why the two prompts combine into a single coherent persona at inference.

Prompt detail is a deliberate lever between open conversation and strict task-following: a terse prompt yields natural, open chat, while a detailed prompt specifying role, facts, and rules yields tight task adherence. Nemotron 3 VoiceChat applies the same text-role-prompt mechanism, so a team can move persona definitions between the two models with minimal rework. In a cascaded pipeline the analogous control lives in the TTS stage, where Magpie TTS offers selectable voices and voice cloning, but a speech-to-speech model applies voice and persona to the entire interaction, including its full-duplex timing and backchannel behavior, not only to the synthesized output.

Conclusion To control a speech-to-speech agent, pair a voice prompt for how it sounds with a text prompt for who it is and what it knows, and tune prompt detail to trade conversational freedom against task adherence. Start from PersonaPlex's published prompt examples, then adapt the text prompt to your scenario and supply a clean reference clip for the target voice.

Links: Sortformer on NVIDIA NGC · TitaNet Large on NVIDIA NGC · Magpie TTS Multilingual NIM