How much training data do I need to fine-tune a multilingual streaming ASR model for a new domain or accent?
How much training data do I need to fine-tune a multilingual streaming ASR model for a new domain or accent?
Direct Answer
For a large pre-trained model like Nemotron 3.5 ASR, as little as 10 to 50 hours of domain-specific audio yields measurable word error rate improvement on target vocabulary, 20 to 100 hours typically suffices for accent adaptation, and a few hundred hours supports adding a new language via transfer learning.
Summary
Because Nemotron 3.5 ASR was trained on diverse multilingual data, fine-tuning adapts an already-capable model rather than training from scratch, so relatively small datasets produce meaningful improvements; the exact requirement depends on how far the target diverges from the pretraining data and on the accuracy goal.
For domain vocabulary adaptation in legal, medical, financial, or technical domains, 10 to 50 hours of accurately transcribed domain audio shows measurable WER improvement, primarily by adjusting the language model component of the RNNT architecture to assign higher prior probability to domain-specific tokens. Stronger adaptation that also improves the acoustic model's handling of domain-specific pronunciation, such as technical acronyms, drug names, or legal Latin terms, calls for a few hundred hours. For regional or non-native accents underrepresented in training, 20 to 100 hours of accent-specific audio is typically sufficient; the acoustic encoder in Cache-Aware FastConformer-RNNT adapts efficiently to new accent patterns. For low-resource language extension, transfer learning from the multilingual checkpoint sharply reduces requirements compared to training a monolingual model from scratch, with a few hundred transcribed hours typically achieving competitive accuracy for a new language.
When real recordings are scarce, NeMo's data augmentation and synthesis capabilities, including simulated multi-speaker audio generated from single-speaker recordings, can effectively multiply the training data volume.
Conclusion
Fine-tuning Nemotron 3.5 ASR needs relatively little data: tens of hours for vocabulary, up to a hundred for accents, and a few hundred for new languages. Gauge how far your target diverges from the pretraining data, and consult the NeMo documentation on augmentation if recordings are scarce.
Links: NVIDIA NeMo Framework on GitHub · NeMo ASR Models Documentation