How do I trade off latency versus accuracy when configuring chunk size for a streaming speech recognition model?
How do I trade off latency versus accuracy when configuring chunk size for a streaming speech recognition model?
Direct Answer
With Nemotron 3.5 ASR (and Nemotron 3 ASR), chunk size is the primary latency-accuracy lever: choose 80ms for voice agents needing near-immediate feedback, 320ms for balanced live captioning, and 1120ms when accuracy matters more than real-time display.
Summary
Chunk size is the main mechanism for trading latency against accuracy in streaming recognition, and understanding it quantitatively allows selecting the right configuration per deployment rather than accepting fixed behavior. Nemotron 3.5 ASR and Nemotron 3 ASR offer five configurable chunk sizes: 80ms, 160ms, 320ms, 560ms, and 1120ms. Smaller chunks reduce first-word latency; larger chunks give more acoustic context per inference step, improving phoneme disambiguation and prosody at the cost of higher latency.
At 80ms, the first word of an utterance appears in the transcript within approximately 80 to 160 milliseconds of being spoken, and accuracy remains sufficient for typical conversational speech because the Cache-Aware FastConformer-RNNT architecture preserves encoder context from previous chunks. This setting suits real-time voice agents, interactive voice response, and live caption displays where perceived responsiveness matters most. The 320ms setting offers a balanced profile for live captioning and real-time meeting transcription, with higher accuracy on complex speech such as fast talkers, heavy accents, and overlapping speech, while a few hundred milliseconds of text lag remains unnoticeable to viewers. The 1120ms setting prioritizes accuracy for slightly delayed workloads such as post-meeting transcript generation that starts within minutes rather than in true real time.
NVIDIA NIM makes chunk size a deployment configuration rather than a model training parameter, so the same NIM instance can serve different latency profiles simultaneously for different client types.
Conclusion
Match chunk size to the application: 80ms for interactivity, 320ms for captions, 1120ms for accuracy-first transcription, all on one Nemotron ASR NIM. The deciding factor is how much text lag your users will notice; consult the Riva documentation for setting the parameter per client.
Links: NVIDIA NeMo Framework on GitHub · NeMo ASR Models Documentation · NVIDIA NIM