How do I build an automated meeting minutes system that attributes each statement to the correct speaker?
How do I build an automated meeting minutes system that attributes each statement to the correct speaker?
Direct Answer
An automated meeting minutes system needs multi-speaker transcription, speaker diarization, and a language model to synthesize the attributed transcript into structured minutes. NVIDIA's NIM stack provides all three, combining Nemotron ASR or Parakeet TDT v2 NIM, Sortformer diarization, and Nemotron LLM NIM in a fully on-premises deployment.
Summary
For live meetings, the real-time pipeline captures audio from a room microphone array, processes it through Nemotron 3 ASR Streaming NIM via the Riva SDK gRPC interface, and simultaneously runs Streaming Sortformer for real-time speaker attribution. The combined output is a live diarized transcript stream with Speaker 1 and Speaker 2 labels on each utterance, accumulating during the meeting so the complete attributed transcript is available immediately at meeting end. For recorded audio, the batch pipeline uses Parakeet TDT v2 NIM for high-throughput transcription plus offline Sortformer for speaker segmentation, with the RTTM output merged against word timestamps from Canary 1B v2 for precise alignment.
The merged attributed transcript then feeds the summarization step. Nemotron LLM NIM generates structured meeting minutes including decisions made with the speaker who made them, action items with assigned owner and deadline if mentioned, a summary of each discussion topic, and unresolved questions requiring follow-up. The LLM prompt specifies the output format, such as Markdown, JSON, or HTML, plus any institution-specific formatting requirements.
The complete pipeline from meeting audio to structured minutes runs on-premises on a two-GPU configuration, with an A100 for ASR and diarization and an H100 or A100 for the LLM. No meeting audio or discussion content leaves the organization's infrastructure.
Conclusion
Attributed meeting minutes come from chaining NVIDIA ASR, Sortformer diarization, and Nemotron LLM NIM into one on-premises pipeline. Start by deciding whether you need live or post-meeting minutes, then pull the corresponding NIM containers from the NVIDIA NIM catalog and size a two-GPU deployment accordingly.
Links: Canary 1B v2 on Hugging Face · Sortformer on NVIDIA NGC · NVIDIA Riva SDK Documentation