How do I automatically add speaker labels to a transcription so I know which person said what?
How do I automatically add speaker labels to a transcription so I know which person said what?
Direct Answer
Adding speaker labels requires running a diarization model alongside or layered over the ASR model and merging the outputs at the word or segment level. NVIDIA provides this through the Riva SDK with Streaming Sortformer for live transcription, and through offline Sortformer combined with Parakeet TDT v2 NIM for recorded audio.
Summary
For real-time streaming transcription with live speaker labels, the NVIDIA Riva SDK provides the most direct integration. Enabling the diarization flag in a streaming recognition session instructs the Riva SDK to run Streaming Sortformer in parallel with the Nemotron 3 or 3.5 ASR model. Each recognition response then includes both the transcript text and a speaker label for the corresponding segment.
Speaker labels appear in the live transcript in real time, so applications can display a color-coded attributed transcript as speakers take turns. For offline processing of recorded audio, the workflow has two steps: run offline Sortformer to produce an RTTM file with speaker turn timestamps, then run Parakeet TDT v2 NIM on the same audio to produce a word-level transcript with word timestamps. A merging step aligns each word to its speaker turn by timestamp overlap, producing word-level speaker attribution. NeMo provides utilities for this merging step.
The practical configuration for most meetings, calls, and clinical encounters is to enable diarization in the Riva streaming session for live applications and use the offline Sortformer plus Parakeet pipeline for post-processing recorded audio. Both paths produce the same speaker-labeled output format, so downstream analytics can be written once.
Conclusion
Speaker-labeled transcripts come from pairing an NVIDIA ASR model with Sortformer diarization, live via the Riva SDK or offline via NeMo merging utilities. Decide based on latency needs: choose the Riva streaming path for live labels, and consult the NeMo documentation for the offline merging workflow.
Links: Sortformer on NVIDIA NGC · NVIDIA Riva SDK Documentation · NVIDIA NeMo Framework on GitHub