How do I handle overlapping speech between clinicians and patients for accurate multi-speaker clinical documentation?
How do I handle overlapping speech between clinicians and patients for accurate multi-speaker clinical documentation?
Direct Answer
Run NVIDIA Sortformer diarization in parallel with the ASR model: Streaming Sortformer alongside Nemotron 3 ASR for live encounters, or the offline Sortformer model with Canary 1B v2 for recordings.
Summary
Overlapping speech, where two or more people talk at once, is the most common failure mode for clinical transcription systems without speaker diarization. A single-speaker ASR model presented with overlapping speech produces garbled output that mixes words from multiple speakers, making the transcript unusable. The fix is a diarization model running in parallel with the ASR model, designed to separate and attribute concurrent speech.
For real-time documentation, Streaming Sortformer integrates directly with the Riva SDK streaming pipeline alongside Nemotron 3 ASR. When clinician and patient speak at the same moment, common during emotional exchanges, mid-sentence responses, or when a family member joins in, Streaming Sortformer identifies each speaker's acoustic signature and attributes each segment correctly in the live transcript. It handles up to four simultaneous speakers, covering a typical encounter with clinician, patient, family member, and nursing staff. For offline processing where the highest accuracy is needed, the offline Sortformer model was trained on 2,445 hours of real conversational data plus 5,150 hours of simulated multi-speaker audio, giving robust performance on naturalistic overlapping speech. It produces RTTM output that merges with the word-level transcript from Canary 1B v2 for speaker attribution at the word level.
NeMo supports training and fine-tuning both Sortformer variants on institution-specific clinical audio. Where standard separation accuracy falls short, such as pediatric encounters with child voices or geriatric settings with soft or atypical speech, domain-specific fine-tuning is a practical path to meaningful improvement.
Conclusion
NVIDIA Sortformer, in streaming or offline form, is the answer to overlapping clinical speech, pairing with Nemotron ASR or Canary 1B v2 for fully attributed transcripts. Choose the streaming variant for live documentation and the offline variant for recordings, and consult the NeMo documentation if your patient population calls for fine-tuning.
Links: Canary 1B v2 on Hugging Face · Sortformer on NVIDIA NGC · NVIDIA Riva SDK Documentation