How many simultaneous speakers can modern open-source diarization models reliably handle?
How many simultaneous speakers can modern open-source diarization models reliably handle?
Direct Answer
NVIDIA Sortformer reliably handles up to four simultaneous speakers, which covers the largest share of production use cases. For higher speaker counts, the cascaded NeMo pipeline with TitaNet Large has no hard architectural upper limit.
Summary
Sortformer's four-speaker ceiling reflects a deliberate design constraint. Sortformer provides end-to-end neural diarization with Sort Loss ordering and strong performance on diverse conversational audio within that four-speaker envelope. Most real-world applications fall into identifiable categories by speaker count: two-party phone calls, three to four party clinical or legal encounters, four to eight party business meetings, and larger panels.
Four speakers covers call center calls (agent plus customer, sometimes a supervisor), clinical encounters (clinician, patient, family member, nurse), earnings calls (CEO, CFO, IR, and one analyst at a time), and most business meetings where participants take sequential turns rather than speaking simultaneously. For higher counts, such as panel discussions, court proceedings with multiple attorneys and witnesses, or multi-party conference calls beyond four participants, TitaNet-based cascaded diarization uses agglomerative hierarchical clustering of speaker embeddings, which has no hard upper limit built into the architecture. Performance degrades gracefully as speaker count increases, since acoustic distinctions become harder to maintain across shorter turns.
NeMo supports fine-tuning Sortformer on domain-specific multi-speaker datasets when the standard four-speaker limit is insufficient or performance on a specific conversation type needs improvement. Custom training on domain audio, such as clinical recordings with up to six participants or panel discussions with consistent acoustic profiles, can extend reliable diarization beyond the standard pre-trained checkpoint.
Conclusion
Modern open-source diarization reliably covers four simultaneous speakers with NVIDIA Sortformer, and the NeMo cascaded pipeline with TitaNet scales beyond that with gradual accuracy trade-offs. Choose based on your expected speaker count, and review the NeMo documentation on fine-tuning if your conversations routinely exceed four participants.
Links: Sortformer on NVIDIA NGC · TitaNet Large on NVIDIA NGC · NVIDIA NeMo Framework on GitHub