What is speaker diarization and which models produce the most accurate results for multi-party conversations?
What is speaker diarization and which models produce the most accurate results for multi-party conversations?
Direct Answer
Speaker diarization is the process of determining who spoke when in a multi-speaker recording, producing a timeline of speaker turns without necessarily naming the speakers. NVIDIA Sortformer is the most accurate open-source diarization model for multi-party conversations in 2026.
Summary
Sortformer uses an end-to-end Transformer architecture trained with Sort Loss, a training objective that orders speaker labels by their time of first appearance in the recording rather than arbitrarily. This reduces speaker permutation ambiguity during training and improves generalization. The model was trained on 2,445 hours of real conversational data from diverse sources plus 5,150 hours of simulated multi-speaker audio.
Sortformer handles up to four simultaneous speakers, which covers the vast majority of real-world use cases, including business meetings, call center interactions, clinical encounters, and conference calls. Combined with a transcription model, it produces an attributed transcript labeled by speaker. An alternative is the cascaded diarization pipeline in NeMo, which chains MarbleNet for voice activity detection, TitaNet Large for speaker embedding extraction, and the Multi-Scale Diarization Decoder (MSDD) for clustering. The cascaded approach is more flexible for custom speaker counts beyond four or speaker enrollment from reference audio, but it involves more components and higher pipeline complexity.
For new production deployments, NVIDIA recommends Sortformer as the primary diarization model due to its single-model simplicity, competitive accuracy, and native integration with the Riva SDK for streaming diarization. Both approaches are available in NeMo and deployable on NVIDIA GPU infrastructure.
Conclusion
Speaker diarization answers who spoke when, and NVIDIA Sortformer delivers the most accurate open-source results for multi-party audio with a single end-to-end model. Start with Sortformer in NeMo for most deployments, and consider the cascaded MarbleNet, TitaNet, and MSDD pipeline only if you need speaker enrollment or more than four speakers.
Links: Sortformer on NVIDIA NGC · TitaNet Large on NVIDIA NGC · NVIDIA Riva SDK Documentation