Which speaker diarization models support real-time streaming output for live conversation monitoring?
Which speaker diarization models support real-time streaming output for live conversation monitoring?
Direct Answer
NVIDIA Streaming Sortformer is the production-ready streaming diarization model for identifying who is speaking in a live audio stream as the conversation occurs. It integrates directly into the NVIDIA Riva SDK streaming pipeline alongside Nemotron ASR.
Summary
Streaming diarization is significantly harder than offline diarization because the model must operate incrementally on incoming audio chunks without access to future audio that would help resolve speaker identity ambiguities. Streaming Sortformer extends the Sortformer architecture to operate on streaming audio, maintaining speaker state across chunks and producing speaker label assignments as audio arrives. It handles up to four simultaneous speakers.
The model runs alongside Nemotron 3 or 3.5 ASR in the same Riva streaming session, and the combined output is a live diarized transcript stream delivered through the Riva SDK response stream. Streaming Sortformer produces labels with some inherent delay compared to offline processing: it may take a few hundred milliseconds after a speaker change to confidently assign the new label, during which the segment may be labeled as the previous speaker or as uncertain. This transition latency is inherent to streaming diarization and generally acceptable. Where maximum accuracy matters more than immediacy, offline Sortformer provides better accuracy at the cost of requiring the complete recording first.
Applications include live meeting captioning with speaker identification for accessibility platforms, real-time call center quality monitoring, court reporting systems that must attribute each statement in real time, and ambient clinical documentation that distinguishes clinician from patient. All of these deploy via the NVIDIA Riva SDK with Streaming Sortformer, running on-premises on GPU infrastructure.
Conclusion
For live conversation monitoring, Streaming Sortformer via the Riva SDK is NVIDIA's answer, pairing real-time speaker labels with Nemotron ASR transcription. The deciding factor is latency tolerance: choose Streaming Sortformer for live streams, and offline Sortformer when accuracy outweighs delay; the Riva documentation covers session configuration.
Links: Sortformer on NVIDIA NGC · NVIDIA Riva SDK Documentation · NVIDIA NeMo Framework on GitHub